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General Guidelines Overview 



Welcome to the Search Quality Rating Program! 

As a Search Quality evaluator, you will work on many different types of rating projects. The General Guidelines 
primarily cover Page Quality (PQ) rating and Block Utility (BU) rating; however, the concepts are also important for 
many other types of rating tasks. 

For brevity, we refer to "Search Quality Evaluators" as "raters" in these guidelines. 




Is this page trustworthy? 



Buy cheap iPhones for cheap - BuvCheaplphones.123 

www.buycheapphones.123/buy-cheap-cheaper 

Browse and buy iPhones on the web's number #1 cheap iPhone market place, 
Register for FREE! Get cheap Iphones here. 
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Part 1 : Page Quality Rating Guideline 



1.0 Introduction to Page Quality Rating 

A Page Quality (PQ) rating task consists of a URL and a series of questions designed to guide your exploration of the 
landing page and the website associated with the URL. Ultimately, the goal of Page Quality rating is to evaluate how 
well the page achieves its purpose. Because different types of websites and webpages can have very different 
purposes, our expectations and standards for different types of pages are also different. 

Here's what you'll need to be a successful Page Quality rater: 

• Your experience using the web as an ordinary user in your rating locale. 

• In-depth knowledge of this guideline. 

• And most importantly — practice doing PQ rating tasks! 

The examples in this guideline are very important. Please view each one. Webpages and websites change rapidly, so 
we use images or "snapshots" of webpages in most of our examples. 

2.0 Understanding Webpages and Websites 

PQ rating requires an in-depth understanding of websites. We'll start with the basics. Along the way, we'll share 
important information about Page Quality rating, so please read through this section even if you are a website expert! 



2.1 Important Definitions 

We will try to avoid using technical jargon, but here are some important definitions: 

A webpage is a document which is part of the World Wide Web. Webpages can be viewed or "visited" with a web 
browser such as Chrome. In the 1990s, webpage content was mostly text and links. Today, webpage content 
includes many forms of media (such as images, videos, etc.) and functionality (such as online shopping features, 
email, calculator functionality, online games, etc.). 

A URL is a character string which your web browser uses to "find" and display a webpage. Page Quality rating doesn't 
require you to have in-depth understanding of the structure of URLs, i.e., you don't need to know the difference 
between host, domain, etc. But if you are interested, see here to read more. 

A website or site is a group of World Wide Web pages usually containing hyperlinks to each other and made available 
online by an individual, company, educational institution, government, or organization. Popular websites include 
Facebook, Wikipedia, Yahoo, YouTube, etc. 

Note: In this guideline, we will use the word "website" to refer to a collection of pages owned and controlled by a single 
entity (individual, business, etc.). But we will also use "website" to refer to major "independent" sections (or hosts) of 
some websites which were created to achieve separate purposes. For example, the Yahoo website is organized into 
different sections (or hosts), such as Yahoo Finance ( finance.vahoo.com) , Yahoo Mail ( mail.vahoo.com) , Yahoo Sports 
( sports.vahoo.com ), etc. Each of these has its own purpose. It's QK to refer to each of these sections as a website; 
for example, the Yahoo Finance website and the Yahoo Sports website. You may also refer to pages on Yahoo 
Finance or Yahoo Sports (even the main entry pages) as belonging to the Yahoo website. 

A homepage of a website is the main page of the site. It is usually the first page that users see when the site loads. 
For example, http://www.apple.com is the homepage of the Apple computer company site, http://www.vahoo.com is 
the homepage of the Yahoo company site, and http://finance.vahoo.com is the homepage of Yahoo Finance. You can 
usually find the homepage of a website by clicking on a "home" link or logo link on subpages of a website. 

A subpage on a website is any page on the site other than the homepage. For example, http://www.apple.com/iphone 
is a subpage on the Apple website, and http://finance.vahoo.com/options is a subpage on the Yahoo Finance website. 
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A webmaster is the person who designs, develops, and/or maintains a website. 

Important; You don't have to be an expert in URLs, webpages, or website design. But you do need to be very 
comfortable exploring websites, both by clicking links and modifying URLs in the address bar of your web browser. 
Become a website detective and explorer! 



2.2 What is the Purpose of a Webpage? 

The purpose of a page is the reason or reasons why the page was created. Every page on the Internet is created for a 
purpose, or for multiple purposes. Most pages are created to be helpful for users. Some pages are created merely to 
make money, with little or no effort to help users. Some pages are even created to cause harm to users. The first step 
in understanding a page is figuring out its purpose. 

Why is it important to determine the purpose of the page for PQ rating? 

• The goal of PQ rating is to determine how well a page achieves its purpose. In order to assign a rating, you 
must understand the purpose of the page and sometimes the website. 

• We have very different standards for different types of pages. By understanding the purpose of the page, 
you'll better understand what criteria are important to consider when evaluating that particular page. 

• Websites and pages should be created to help users. Websites and pages which are created with intent to 
harm users, deceive users, or make money with no attempt to help users, will receive a very low PQ rating. 
More on this later. 

As long as the page is created to help users, we will not consider one particular purpose or type of page to be higher 
quality than another. For example, encyclopedia pages are not necessarily higher quality than humor pages. 

Important: There are highest quality and lowest quality webpages of all different types and purposes: shopping pages, 
news pages, forum pages, video pages, pages with error messages, PDFs, images, gossip pages, humor pages, 
homepages, and all other types of pages. Each of these types of pages can have highest quality ratings and lowest 
quality ratings. 

Qur expectations are different for different kinds of websites. Imagine a website called "Uncle Alex's Family Photos" (a 
hypothetical High quality example) created with much time and effort for the purpose of sharing photos with relatives. 
Compare that to the website of a multimillion dollar corporation which exists to sell products. Page quality rating is not 
"one size fits all." You have to think about the purpose of the page and website. 

Common helpful page purposes include (but are not limited to): 

• To share information about a topic. 

• To share personal or social information. 

• To share pictures, videos, or other forms of media. 

• To express an opinion or point of view. 

• To entertain. 

• To sell products or services. 

• To allow users to post questions for other users to answer. 

• To allow users to share files or to download software. 



Here are a few examples where it is easy to understand the purpose of the page: 



Type of Page 


Purpose of the Page 


News website homepage 


To display news. 


Shoppina paqe 


To sell or give information about the product. 


Video paqe 


To allow users to watch a video. 


Currency converter page 


To calculate equivalent amounts in different currencies. 
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Here are two examples of helpful pages where the purpose of the page is not as obvious: 



Page with a Non- 
Obvious Purpose 


Discussion 


Christopher Columbus 
Paqe Example 


This page looks as though the purpose is to share factual information, but the page starts with the text 
"Christopher Columbus was born in 1951 in Sydney, Australia." This is obviously inaccurate! Was this 
page created to help users or to trick and confuse users? 

In this case, exploring the website can help us understand the purpose of the page. This website was 
built by educators to teach about interpreting information found on the Internet. 

After reading about the website on the About This Site page, it should be clear that the purpose of the 
page is to serve as an educational tool. The information on the page is deliberately inaccurate so that it 
can be used as an example of misinformation on the Internet. This page and website do have a helpful 
and beneficial purpose. 


OmNomNomNom 
Paae Example 


At first glance, this page may seem pointless or strange. However, it is a page from a humorous site 
that encourages users to post photos with mouths drawn on them. The purpose of the page is humor or 
artistic expression. This page has a helpful or beneficial purpose. 

Even thouqh the About paae on this website is not very helpful, the website explains itself on its FAQ 
page. 



2.3 Your Money or Your Life (YMYL) Pages 

Some types of pages could potentially impact the future happiness, health, or wealth of users. We call such pages 
"Your Money or Your Life" pages, or YMYL. The following are YMYL pages: 

• Shopping or financial transaction pages: webpages which allow users to make purchases, transfer money, pay 
bills, etc. online (such as online stores and online banking pages). 

• Financial information pages: webpages which provide advice or information about investments, taxes, 
retirement planning, home purchase, paying for college, buying insurance, etc. 

• Medical information pages: webpages which provide advice or information about health, drugs, specific 
diseases or conditions, mental health, nutrition, etc. 

• Legal information pages: webpages which provide legal advice or information on topics such as divorce, child 
custody, creating a will, becoming a citizen, etc. 

• Other: there are many other topics which you may consider YMYL, such as child adoption, car safety 
information, etc. Please use your judgment. 

We have very high Page Quality rating standards for YMYL pages because low quality YMYL pages could potentially 
negatively impact users' happiness, health, or wealth. 

2.4 Understanding Webpage Content 

All of the content on a webpage can be classified as one of the following: Main Content (MC), Supplementary Content 
(SC), or Advertisements/Monetization (Ads). In order to understand the purpose of a webpage and do PQ rating, you 
will need to be able to distinguish among these different parts of the page. 

Webpage design can be complicated, so make sure to click around and explore the page. See what kind of content is 
behind the tabs and test out the interactive page features. 

2.4.1 Identifying the IVIain Content (IVIC) 

Main Content is any part of the page that directly helps the page achieve its purpose. MC can be text, images, videos, 
or page features such as calculators, games, etc. MC can be user generated content such as videos, reviews, 
articles, etc. which users have added or uploaded to the page. Note that tabs on some pages lead to even more 
information, for example, customer reviews. In some cases, you can consider the content under or behind tabs to be 
part of the MC of the page. Webmasters directly create and/or control the MC of the page. 
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Type of Page and Purpose 


MC Highlighted in Yellow 


News website homepage: the purpose is to display news. 


MC - News Homepage 


News article page: the purpose is to display a news article. 


MC - News Article 


Store product page: the purpose is to sell or give information about the product. 
• Content behind the Reviews, Shipping, and Safety Information tabs are considered to be 
part of the MC 


MC - ShoDDinq Paae 


Video page: the purpose is to allow users to view a video. 


MC - Video Paqe 


Currency converter page: the purpose is to calculate equivalent amounts in different currencies. 


MC - Currency Converter 


Blog post page: the purpose is to display a blog post. 


MC - Bloq Post Paqe 


Search engine homepage: the purpose is to allow users to enter a query and search the Internet. 


MC - Search Enqine Homeoaqe 


Bank login page: the purpose is to allow users to log in to bank online. 


MC - Bank Loqin Paqe 



2.4.2 Identifying the Supplementary Content (SC) 



Supplementary Content contributes to a good user experience on the page, but does not directly help the page 
achieve its purpose. SC is created by webmasters and is an important part of the user experience. One common type 
of SC is navigation links which allow users to visit other parts of the website. Note that content behind tabs may be 
considered part of the SC of the page. 

Sometimes the easiest way to identify SC is to look for the parts of the page which are not MC or Ads. 



Type of Page and Purpose 


SC Highlighted in Blue 


News website homepage: the purpose is to display news. 


SC - News Homeoaqe 


News article page: the purpose is to display a news article. 


SC - News Article 


Store product page: the purpose is to sell or give information about the product. 


SC - ShoDDinq Paqe 


Video page: the purpose is to allow users to view a video. 


SC - Video Paqe 


Currency converter page: the purpose is to calculate equivalent amounts in different currencies. 


SC - Currency Converter 


Blog post page: the purpose is to display a blog post. 


SC - Bloq Post Paqe 


Search engine homepage: the purpose is to allow users to enter a query and search the Internet. 


SC - Search Enqine Homepaqe 


Bank login page: the purpose is to allow users to log in to bank online. 


SC - Bank Loqin Paqe 



2.4.3 Identifying Advertisements/Monetization (Ads) 



Advertisements/Monetization (Ads) is content and/or links that are displayed for the purpose of monetizing (making 
money from) the page. There are several different ways to monetize a webpage, including advertisements and affiliate 
programs. See here for more information on website monetization. 

The most common type of monetization is advertisements. Ads may be labeled as "ads," "sponsored links," 
"sponsored listings," "sponsored results," etc. Usually, you can click on the links or mouse over the content to 
determine whether they are Ads. Ads may change when you reload the page, and different users may see different 
Ads on the same page. 

Webmasters can choose to display Ads on their page (for example by joining an advertising network ), but they may not 
always directly control the content of the Ads. However, we will consider a website responsible for the overall quality 
of the Ads displayed. 

Important: For the purpose of this guideline, we will consider monetized links of any type to be "Ads." See here for 
different types of website monetization. 

Important: Without advertising and monetization, some webpages could not exist because it costs money to maintain 
a website and create high quality content. The presence or absence of Ads is not by itself a reason for a High or Low 
quality rating. 
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Type of Page and Purpose 


Ads Highlighted in Red 


News website homepage: the purpose is to display news. 


Ads - News Homeoaqe 


News article page: the purpose is to display a news article. 


Ads - News Article 


Video page: the purpose is to allow users to view a video. 


Ads - Video Page 


Currency converter page: the purpose is to calculate equivalent amounts in different currencies. 


Ads - Currency Converter 


Blog post page: the purpose is to display a blog post. 


Ads - Bloq Post Page 


Store product page: the purpose is to sell or give information about the product. 


No ads 


Search engine homepage: the purpose is to allow users to enter a query and search the Internet. 


No ads 


Bank login page: the purpose is to allow users to log in to bank online. 


No ads 



2.4.4 Summary of the Parts of the Page 

Let's put it all together. 

Main Content (MC) is any part of the page that directly helps the page achieve its purpose. MC is (or should be!) the 
reason the page exists. The quality of the MC plays a very large role in the Page Quality rating of a webpage. 

Supplementary Content (SC) is also important. SC can help a page better achieve its purpose or it can detract from 
the overall experience. 

Many pages have advertisements/monetization (Ads). Without advertising and monetization, some webpages could 
not exist because it costs money to maintain a website and create high quality content. The presence or absence of 
Ads is not by itself a reason for a High or Low quality rating. 

On some pages, reviews may be considered MC, and on other pages they may be considered SC. Use your best 
judgment and think about the purpose of the page. 



Do not worry too much about identifying every little part of the page. Carefully think about which parts of the page are 
the MC. Next, look for the Ads. Anything left over can be considered SC. 



Type of Page and Purpose 


lUIC, SC, and Ads Highlighted 


News website homepage: the purpose is to display news. 


Summary - News Homeoaae 


News article page: the purpose is to display a news article. 


Summary - News Article 


Store product page: the purpose is to sell or give information about the product. 


Summary - Shopping Page 


Video page: the purpose is to allow users to view a video. 


Summary - Video Page 


Currency converter page: the purpose is to calculate equivalent amounts in different currencies. 


Summary - Currency Converter 


Blog post page: the purpose is to display a blog post. 


Summary - Bloq Post Page 


Search engine homepage: the purpose is to allow users to enter a query and search the Internet. 


Summary - Search Engine HP 


Bank login page: the purpose is to allow users to log in to bank online. 


Summary - Bank Loqin Paqe 



2.5 Understanding the Website 

Pages often make more sense when viewed as part of a website. Some of the considerations in Page Quality rating 
are based on the website the page belongs to. 

In order to understand a website, look for information about the website on the website itself. Websites are usually 
very eager to tell all about themselves! 

You must also look for reputation information about the website. We need to find out what outside, independent 
sources say about the website. When there is disagreement between what the website says about itself and what 
independent sources say about the website, we'll trust the independent sources. See Section 2.7 for more 
information. 
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2.5.1 Finding the Homepage 



The homepage of a website usually contains or has links to important information about the website. Webmasters 
usually make it easy to get to the homepage of the website from any page on the site. 

Here's how to find the homepage of a website: 

• Examine the landing page of the URL in your PQ rating task. 

• Find and click on the link labeled "home" or "main page." 

• Having trouble finding it? Try using "Ctrl-F" ("command-F" on a Mac) to search the page for the text "home" or 
"main." You may also try clicking on the website logo, which is usually at the top of the page. 

Sometimes, you may be given a webpage or website that appears to have no navigation links, no homepage link, and 
no logo or other means to find the homepage. Even some High or Highest quality pages lack a way to navigate to the 
homepage. If you can't find a link to the homepage, modify the URL by removing everything to the right of ".com," 
".org," ".net," ".info," etc. and refresh the page. 

Occasionally, your rating task will include a URL for which there are two or more justifiable "homepage" candidates. 
For example, you may not be sure whether the homepage of the URL http://finance.vahoo.com/news/cateqorv-stocks 
is http://finance.vahoo.com or http://www.vahoo.com . 

Important: When you have more than one homepage "candidate," please use whichever one offers the most 
information about the specific webpage in the URL rating task. Use your judgment. The goal is to understand the 
webpage and the website(s) it is associated with, not find the one unique, correct homepage. 

In the following examples, we have included the URL of the page to be evaluated in the rating task, as well as the URL 
of its associated homepage. We have also included an image that shows where to click on the landing page to 
navigate to the homepage. In the image, you will see a red box around the link or the logo you would click to navigate 
to the homepage. 



URL of the Task Page 


Homepage of the Website 


Image that shows where to click to get to 
the homepage 


http://www.willlams- 
sonoma.com/Droducts/s 
hun-Dremier-7-Diece- 
knife-block-set 


httD://www. williams-sonoma.com 


Williams-Sonoma Homepaqe 

WILLIAMS-SONOMA 

This "WILLIAMS-SONOIVIA" logo shown in the 
upper left part of the page is clickable and 
takes users to the homepage of the website. 


httD://answers.vahoo.co 
m/auestion/index; vlt=A 
nAYEU1fED6nca1iRCF 
v30kk5XNG; vlv=3?aid 
=20091214193523AAQ 


httD://answers.vahoo.com 

In this case, we will consider 
httD://answers.vahoo.com the homepage, rather than 
http://www.vahoo.com. Why? Because clickinq on 
the loqo takes the user to http://answers.vahoo.com. 
In addition, http;//answers.vahoo.com has information 


Specific Yahoo Answers Paqe 

"VXIiOOr* ANSWERS 

This "YAHOO ANSWERS" logo in the upper 
left part of the page is clickable and takes 
users to the homepage of the website. 


qHQS 


about the Yahoo Answers website. It is very difficult 
to find specific information about 
htto;//answers.vahoo.com on the 
http://www.vahoo.com homepaae. 
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URL of the Task Page 


Homepage of the Website 


Image that shows where to click to get to 
the homepage 


httpV/hms. harvard.edu/ 
about-hms/facts-fiaures 


http://hms.harvard.edu 

In this case, we will consider the Harvard Medical 
School oaae at httD://hms.harvard.edu to be the 
homepaae, rather than htto://www.harvard.edu 
(which is the homepage of Harvard University). 
Clicking the logo at the top of 
http://hms.harvard.edu/about-hms/facts-fiqures takes 
users to http://hms.harvard.edu, not to 
http://www.harvard.edu. 


Harvard IVIedical School Facts and Figures 
Paqe 

^ HARVARD 

MEDICAL SCHOOL 

This "Harvard IVIedical School" logo in the 
upper left part of the page is clickable and 
takes users to the homepage of the Harvard 






Medical School website. 




http://www.facebook.com 


Oprah Winfrey's Facebook Paqe 


httDs://www.facebook.c 


In this case, we will consider 
http://www.facebook.com the homepaae. Clickinq 
the "facebook" logo at the top of 
https://www.facebook.com/oorahwinfrev takes users 
to http://www.facebook.com 


1 facebook 


om/oorahwinfrev 


The "facebook" logo in the upper left part of 
the page is clickable and take users to the 
homepage of the Facebook website. 






"Arab Women" article on the Cornell Library 
website 


httD://www.librarv. cornel 
I.edu/colldev/mideast/ar 
bwomn.htm 


http://www.librarv.cornell.edu/ 

This is the library website of Cornell University. 


This page does not have any links to the 
homepage of the Cornell University Library 
website. The only way to get to the homepage 
is to modify the URL by removing everything to 
the right of "library.cornell.edu": 

http://www.library.cornell.edu/colldev/mideast/ 
arbwomn.htm 


https://www.kernel.ora/p 
ub/linux/utils/boot/syslin 
ux/ 

Note: The rating for this 
page is discussed in 
Section 5.4 on 
examples of Highest 
quality pages 


http://www.kernel.ora 

This is the website of an organization which helps 
Linux users. Limited navigation links is not a problem 
for this technically savvy audience. 


Kernel.orq Directory Paqe 

This page does not have any links to the 
homepage of the Kernel website. The only 
way to get to the homepage is to modify the 
URL by removing everything to the right of 
"kernel.org": 

https://www.kernel.org/pub/linux/utils/boot/sysli 



2.5.2 Finding Who is Responsible for the Website and Who Created the Content on the Page 

Every page belongs to a website, and it should be clear: 

• Who (what individual, company, business, foundation, etc.) is responsible for the website. 

• Who (what individual, company, business, foundation, etc.) created the content on the page you are 
evaluating. 
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Websites are usually very clear about who created the content on the page. There are many reasons for this: 

• Commercial websites may have copyrighted material they want to protect. 

• Businesses want users to know who they are. 

• Artists, authors, musicians, and other original content creators usually want to be known and appreciated. 

• Foundations often want support and even volunteers. 

• High quality stores want users to feel comfortable buying online. 

Most websites have "contact us" or "about us" or "about" pages which provide information about who owns the site. 
Many companies have an entire website or blog devoted to who they are and what they are doing, what jobs are 
available, etc. Google and Marriott are both examples of this, and there are many others: 

• Google Official Blog 

• Marriott Blog 

• Southwest Airlines Blog 

• Dell Blog 

Often a business or organization is responsible for the content of a website, not an individual person. The IBM 
Corporation is responsible for the content on ibm.com. The Cleveland Clinic is responsible for the content on 
clevelandclinic.org. An individual is not responsible for the content on these websites, even though many individuals 
contributed to creating and maintaining the content. In these cases, we will view the business or organization as 
responsible for the content on every single page, as well as maintenance of the website. 

On some websites, users create the MC of many pages, while the business or organization itself maintains the 
website. For these websites, you must look at each page to determine the author(s) or creator(s) of the content on 
that page. For example, the company Facebook is responsible for the Facebook website, but individuals create the 
content on their personal Facebook pages. Other websites with user-generated content include YouTube, Twitter, 
other social networking websites, article publishing websites, O&A websites, forums, etc. 

Finally, there are some websites which show licensed or syndicated content. This means that the website has paid 
money or has some business relationship with the creator of the content. In these cases, we will consider the website 
itself to be responsible for the licensed or syndicated content, even if it wasn't created by the website. 



2.5.3 Finding About Us, Contact Information, and Customer Service Information 

Many websites are interested in communicating with their users. There are many reasons that users might have for 
contacting a website, from reporting problems such as broken pages, to asking for content removal. Many websites 
offer multiple ways for users to contact the website: email addresses, phone numbers, physical addresses, etc. 
Sometimes, this contact information is even organized by department and provides the names of individuals to contact. 
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HUNTING & FISHING LUGGAGE HOME SALE 

Bifls Gift Cards Blog ParkFindEr Videos L.L.Bean Visa 



Ordering Informalion 
Order Status 
My Account 
Shipping InfO'tmation 
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Sizing S Fit 
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Services 
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Wish List 
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Outdoor Discovery Schools 
International help 
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Heipfui Links 

100% Guaranteed 
Contacl Us 
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Product Recalls 
Your Privac7 
AC out Security 




100 YEARS o/OUTSTANDIHG CUSTOMER SERVICE* 

Proi/iding great customer service is not an option - it's been the guiding principle 
beliind our business for I'-OO years Have questions before, during or after you 
place your order? We've got answers. 



To get slaied select a topic at left or choose a link below. 



► Forgot Your Password 
* Stze Charts 

CONTACT US: O Plio^e 



► Order Tract^ing 

t Returns and E/changes 

Call Me W Live H 



► Coupons. Gift Cards and Promos 

► Monogramming 



Ordering 1nformatk>n Shipping fnfonnatiDU 

Placing an Order at IIDean.com U3 a Canada 

More Ways to Order international 

PaymentOpBons Freigtil Delivery 

Coupons, GiftCardsS Free snip-to-Store 

Promotions Shipping to Multiple Addres; 



Morogramm 

Engraving 

Repairs 



Order status 
Order Tracking 
Purchase History 
Backorders 

My Acu>unt 

Forgot VourPassword 
How to Create an Account 
Email Preferences 



Returns S Exchanges 

Easy Returns & Exchanges 
QuIckExchange 
Freight Returns 

SiLrng 8. Fit 

Men's, Women's 8i Kids' & 
Bedding Size Chans 
L.L.Bean Signature Size Clarts Catalog Mailing 
Alterations Preferences 



Gift Options 
Gift Boxing 
GItt Message 

CatabgE 

Request a Catalog 
Shop Catalogs Online 



11 More Contact Info 

Outdoor Discovery Schools 
iilakmg a Reservation 
Schedule Changes and 
Cancellations 



Intemational itelp 

Duties and Taxes 
Import/Export Restrictions 
Shipping Items Uninsured 

Technical lnfomiatk>n 
WeD Browsers 
Coolaes 
MoOlle Devices 



The types and amount of contact information needed depend on tine type of website. Contact information and 
customer service information are extremely important for websites tinat Inandle money, sucin as stores, banks, credit 
card companies, etc. Users need a way to ask questions or get Inelp winen a problem occurs. 

For shopping websites, we'll ask you to do some special checks. Look for contact information — including the store's 
policies on payment, exchanges, and returns. Sometimes this information is listed under "customer service." 

Some kinds of websites need fewer details and a smaller amount of contact information for their purpose. For 
example, humor websites may not need the level of detailed contact information we would expect an online banking 
website to have. 

Occasionally, you may encounter a website with a legitimate reason for anonymity. For example, personal websites 
may not include personal contact information such as an individual's home address or phone number. 

To find contact or customer service information for a website, start with the homepage. Look for a "contact us" or 
"customer service" link. Explore the website if you cannot find a "contact us" page. Sometimes you will find the 
contact information on a "corporate site" link or even on the company's Facebook page. Be a detective! 



2.6 Website Maintenance 

Webmasters are responsible for updating and maintaining sites they create. Most websites add or change content 
over time. Web browsers, such as Chrome, update with new versions. Webmasters need to make sure their websites 
function well for users as web browsers change. 

How can you tell that a website is being maintained and cared for? Poke around: Links should work, images should 
load, content should be added and updated over time, etc. 

Exercise caution relying on dates: Some webpages automatically display the current date. Rather than just looking for 
a recent date, search for evidence that effort is being made to keep the website up to date and running smoothly. 

Finally, the types of updates needed depend on the purpose of the website and type of content. We expect news 
websites to add articles very frequently and to date each article. Typically, published news article content doesn't 
change (unless to correct for errors), but new articles are added. On other websites, individual pages created on a 
topic are updated as new information becomes available. Wikipedia is an example of this. For these kinds of sites, we 
would expect individual pages to be updated as information changes. 
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2.7 Website Reputation 

A website's reputation is based on tine experience of real users, as well as the opinion of people who are experts in the 
topic of the website. 

Many websites are eager to tell users how great they are. But for Page Quality rating, you must also look for outside, 
independent reputation information about the website. When the website says one thing about itself, but reputable 
external sources disagree with what the website says, trust the external sources. 

Your job is to truly evaluate the Page Quality of the site, not just blindly accept information on one or two pages of the 
website. Be skeptical of claims that websites make about themselves. 

Keep in mind that websites often represent real companies, organizations, and other entities. Therefore, reputation 
research applies to both the website and the actual company, organization, or entity that the website is representing. 

Note: Some webmasters have read these rating guidelines and have included information on their sites to influence 
your Page Quality rating! 



2.7.1 Reputation Research 

Use reputation research to find out what real users, as well as experts, think about a website. Look for reviews, 
references, recommendations by experts, news articles, and other credible information created/written by individuals 
about the website. 

Stores frequently have user ratings, which can help you understand a store's reputation based on the reports of people 
who actually shop there. We consider a large number of positive user reviews as evidence of positive reputation. 

Many other kinds of websites have reputations as well. For example, you might find that a newspaper website has 
won journalistic awards. Prestigious awards, such as the Pulitzer Prize award, are strong evidence of very positive 
reputation. 

When a high level of authoritativeness or expertise is needed, the reputation of a website should be judged on what 
expert opinions have to say. Recommendations from expert sources, such as professional societies, are strong 
evidence of very positive reputation. 

Reputation research is necessary for all websites you encounter. Do not just assume websites you personally use 
have a good reputation. Please do research! You might be surprised at what you find. 



2.7.2 Sources of Reputation Information 

Look for information written by a person, not statistics or other machine-compiled information. News articles, 
Wikipedia articles, blog posts, magazine articles, forum discussions, and ratings from independent organizations can 
all be sources of reputation information. Look for independent, credible sources of information. 

Sometimes, you will find information about a website which is not related to its reputation. For example, pages with 
information about Internet traffic to the website do not provide evidence of positive or negative reputation. You can 
ignore this information since it's not helpful for Page Quality rating. 



2.7.3 Customer Reviews of Stores/Businesses 

Customer reviews can be helpful for assessing the reputation of a store or business. However, you should interpret 
these reviews with care, particularly if there are only a few. Be skeptical of both positive and negative user reviews. 
Anyone can write them, including the creator of the website or someone the store or business hires for this purpose. 
See here for a New York Times article on fake reviews and here for a Guardian article on fake reviews. 
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When interpreting customer reviews, try to find as many as possible. Any store or website can get a few negative 
reviews. TInis is completely normal and expected. Large stores and companies have thousands of reviews and most 
receive some negative ones. 

It is also important to read the reviews because the content of the reviews matter, not just the number. Credible, 
convincing reports of fraud and financial wrongdoing is evidence of extremely negative reputation. A single encounter 
with a rude clerk or the delayed receipt of a single package should not be considered negative reputation information. 
Please use your judgment. 

2.7.4 How to Search for Reputation Information 

Here is how to research the reputation of the website: 

1 . Identify the "homepage" of the website. For example, for the IBM website, ibm.com is the homepage. 

2. Using ibm.com as an example, try one or more of the following searches on Google: 

• [ibm -site:ibm.com]: A search for IBM which excludes pages on ibm.com. 

• ["ibm.com" -site:ibm.com]: A search for "ibm.com" which excludes pages on ibm.com. 

• [ibm reviews -site:ibm.com] A search for reviews of IBM which excludes pages on ibm.com. 

• ["ibm.com" reviews -site:ibm.com]: A search for reviews of "ibm.com" which excludes pages on 
ibm.com. 

3. Look for articles, reviews, forum posts, discussions, etc. written by people about the website. For businesses, 
there are many sources of reputation information and reviews. Here are some examples: Yelp , Better 
Business Bureau (a nonprofit organization that focuses on the trustworthiness of businesses and charities), 
Amazon , and Google Product Search . You can try searching on specific sites to find reviews. For example, 
you can try [ibm site:bbb.org] or ["ibm.com" site:bbb.org]. 

Note: You will sometimes find high ratings on the Better Business Bureau (BBB) website because there is very 
little data on the business, not because the business has a positive reputation. However, low ratings on BBB 
are usually the result of multiple unresolved complaints. Please consider very low ratings on the BBB site to 
be evidence for a negative reputation. 

4. See if there is a Wikipedia article or news article from a well-known news site. Wikipedia can be a good 
source of information about companies and organizations. For example, try [ibm site:en. wikipedia. org] or 
["ibm.com" site:en. wikipedia. org]. News articles and Wikipedia articles can help you learn about a company 
and may include information specific to reputation, such as awards and other forms of recognition. Note that 
some Wikipedia articles include a message warning users that there are disagreements on some of the 
content, or that the content may be outdated. This may be an indication that additional research is necessary. 



Here are some examples of reputation information: 



Website 


Reputation Information About tfie Site 


Description 


annualcreditreDort.com 


Search results for fannualcreditreDort.com 
-siteiannualcreditreDort.coml 

Wikioedia article about 


Positive reputation information: Users in the U.S. can 
obtain free credit reports on this website by providing their 
Social Security Number. Notice that this Wikipedia article 
tells us that "AnnualCreditReport.com is the only federally 
mandated and authorized source for obtaining a free credit 
report." 


annualcreditreDort.com 

Wall Street Journal article about 
annualcreditreDort.com 
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Website 


Reputation Information About the Site 


Description 


clevelandclinicorq 


Search results for [clevelandclinic.orql 

Wikipedia article about clevelandclinicorq 

US News & World Report article about the 
best hospitals in the U.S. 


Positive reputation information: According to Wikipedia, 
the Cleveland Clinic "is currently regarded as one of the top 
4 hospitals in the United States as rated by U.S. News & 
World Report," which you will also find in the article on the 
best hospitals in the U.S. Users can trust medical 
information on this website. 


csmonitor.com 


Search results for [csmonitor.com - 
site:csmonitor.coml 

Wikipedia article about The Christian 


Positive reputation information: Notice the highlighted 
section in the Wikipedia article about The Christian Science 
Monitor newspaper, which tells us that the newspaper has 
won seven Pulitzer Prize awards. From this information, we 
can infer that the csmonitor.com website has a positive 
reputation. 




Science Monitor 


llbean.com 


Search results for [llbean.com reviews - 
site:llbean.coml 

LL Bean positive reviews 1 


Positive reputation information: From the numerous 
positive reviews, we can infer that llbean.com has a positive 
reputation. 


LL Bean positive reviews 2 


LL Bean positive reviews 3 


kernel. orq 


Search results for [kernel.orq - 
site:kernel.orql 

Wikipedia article about kernel.orq 


Positive reputation information: We learn in the Wikipedia 
article that "Kernel.org is a main repository of source code 
for the Linux kernel, the base of the popular Linux operating 
system. It makes all versions of the source code available 
to all users. It also hosts various other projects, like Google 
Android. The main purpose of the site is to host a repository 
for Linux kernel developers and maintainors of Linux 
distributions." 


Site sellinq children's 
iunqle qvm 


Search to find reputation information 


Extremely negative reputation information: This business 
has a BBB rating of F. There is a news article about 
financial fraud. There are many reviews on websites 
describing users sending money and not receiving anything. 


Search to find reviews 
BBB neqative review 
TrustLink neqative reviews 
Neqative news article 


Site sellinq products 
related to eyewear 


Search to find reputation information 
BBB paqe 
Wikipedia article 


Extremely negative/malicious reputation information: 

This website engaged in criminal behavior such as 
physically threatening users. 


New York Times article 
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Website 


Reputation Information About the Site 


Description 


Oraanization servina 
the hospitalized 
veteran communitv 


Search to find scams related this 
oraanization 

Neqative review 1 

Neqative review 2 

Neaative review 3 

Neqative review 4 


Extremely negative reputation information: There are 
many detailed negative articles on news sites and charity 
watchdog sites about this organization describing fraud and 
financial mishandling. 



2.7.5 What to Do When You Find No Reputation Information 

You should expect to find reputation information for large businesses and websites of large organizations. 

Frequently, you will find little or no information about the reputation of a website for a small organization. This is not 
indicative of positive or negative reputation. Many small, local businesses or community organizations have a small 
"web presence" and rely on word of mouth, not online reviews. For these smaller businesses and organizations, lack 
of reputation should not be considered an indication of low page quality. 



3.0 Overall Page Quality Rating Scale 

The Overall Page Quality Rating Scale offers five rating options: Lowest, Low, Medium, High, and Highest. 

On Page Ouality rating tasks, you will use the Page Ouality sliding scale (slider) to assign the overall PO rating. The 
slider looks like this: 



Lowest Lowest+ Low Lo»h- Medium MeAmih- High l-M"- Highest 



You may also use the in-between ratings of Lowest*, Low+, Medium+, and High+. Please interpret the "+" as "+ Vi," 
meaning that the Lowest* rating is halfway between Lowest and Low, the Low+ rating is halfway between Low and 
Medium, etc. 

In the following sections, you will learn about characteristics of Lowest, Low, Medium, High, and Highest quality 
pages. 



4.0 High Quality Pages 

In this section, we will describe characteristics of High quality pages and give many examples. Examples help 
calibrate your ratings, so please review each one. 



4.1 Characteristics of High Quality Pages 

High quality pages are satisfying and achieve their purpose well. High quality pages exist for almost any purpose, 
from giving information to making you laugh. 
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What makes a High quality page? A High quality rating requires at least one of the following high quality 
characteristics: 



• A satisfying amount of high quality MC. 

• The page and website are expert, authoritative, and trustworthy for the topic of the page. 

• The website has a good reputation for the topic of the page. 

In addition, the page and website should have most of the following: 

• A satisfying amount of website information, for example. About Us information. Contact or Customer Service 
information, etc. 

• SC which contributes to a satisfying user experience on the page and website. 

• Functional page design which allows users to easily focus on MC and use SC as desired. 

• A website which is well cared for and maintained. 



4.2 A Satisfying Amount of High Quality IVIain Content 

The quality of the MC is one of the most important considerations in Page Quality rating. For all types of webpages, 
creating high quality MC takes a significant amount of at least one of the following: time, effort, expertise, and 
talent/skill. 

For each page you evaluate, spend a few minutes examining the MC before drawing a conclusion about it. Read the 
article, watch the video, examine the pictures, play with the calculator, play the online game, etc. Remember that MC 
also includes page features and functionality, so test the page out. For example, if the page is a product page on a 
store website, put at least one product in the cart to make sure the shopping cart is functioning. If the page is an 
online game, spend a few minutes playing it. 

The purpose of the page will help you determine what high quality content means for that page. For example. High 
quality encyclopedia articles should be factual, accurate, clearly written, and comprehensive. High quality shopping 
content should allow you to find the products you want and to purchase the products easily. High quality humor 
content should be entertaining. 

The amount of content necessary for the page to be satisfying depends on the topic and purpose of the page. A High 
quality page on a broad topic with a lot of available information will have more content than a High quality page on a 
more narrow topic. Here are some examples of pages with a satisfying amount of high quality MC. 



Webpage 


Discussion 


Siberian Husky 
Information Paae 


The Siberian Husky (a breed of dog) is a narrow topic. Although this encyclopedia landing page has less MC 
than some encyclopedia pages on broader topics, it has a satisfying amount of clearly written, high quality 
MC. 


Kitchen Stand 
Mixer ShoDDina 
Paae 


This shopping page on a reputable shopping website has a satisfying amount of high quality MC. The page 
provides the manufacturer's product specs, as well as original product information, over 90 user reviews, 
shipping and returns information, multiple images of the product, etc. Note: Some of the MC is behind links on 
the page ("item details," "item specifications," "guest reviews," etc.). Even though you have to click these links 
to see the content, it is still considered MC. 


Movie Review 
Paae 


This movie review written by a movie critic has a satisfying amount of high quality MC. Time, effort, and 
talent/skill when into writing this movie review. 
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4.3 A High Level of Expertise/ Authoritativeness/Trustworthiness (E-A-T) 

High quality pages and websites need enougin expertise to be autlnoritative and trustwortlny on tineir topic. Keep in 
mind tinat tinere are "expert" websites of all types, even gossip websites, fashion websites, humor websites, forum and 
Q&A pages, etc. In fact, some types of information are found almost exclusively on forums and discussions, where 
community of experts can provide valuable perspectives on specific topics. 

• High quality medical advice should come from people or organizations with appropriate medical expertise or 
accreditation. High quality medical advice or information should be written or produced in a professional style 
and should be edited, reviewed, and updated on a regular basis. 

• High quality financial advice, legal advice, tax advice, etc., should come from expert sources and be 
maintained and updated. 

• High quality advice pages on topics such as home remodeling (which can cost thousands of dollars) or advice 
on parenting issues (which can impact the future happiness of a family) should also come from "expert" 
sources which users can trust. 

• High quality pages on hobbies, such as photography or learning to play a guitar, also require expertise. 

Some topics require less formal expertise. Many people write extremely detailed, helpful reviews of products or 
restaurants. Many people share tips and life experiences on forums, blogs, etc. These ordinary people may be 
considered experts in topics where they have life experience. If it seems as if the person creating the content has the 
type and amount of life experience to make him or her an "expert" on the topic, we will value this "everyday expertise" 
and not penalize the person/page/website for not having "formal" education or training in the field. 

It's even possible to have everyday expertise in YMYL topics. For example, there are forums and support pages for 
people with specific diseases. Sharing personal experience is a form of everyday expertise. Consider this example . 
Here, forum participants are telling how long their loved ones lived with liver cancer. This is an example of sharing 
personal experiences (in which they are experts), not medical advice. However, specific medical information and 
advice (rather than descriptions of life experiences) should come from doctors or other health professionals. 

Think about the topic of the page. What kind of expertise is required for the page to achieve its purpose well? The 
standard for expertise depends on the topic of the page. 



4.4 Positive Reputation 

Reputation is an important consideration when using the High rating. While a page can merit the High rating with no 
reputation, the High rating cannot be used for any website that has a convincing negative reputation. A very positive 
reputation can be a reason for using the High rating for an otherwise IVIedium page. 



4.5 Helpful Supplementary Content 

Supplementary Content can be a large part of what makes a High quality page very satisfying for its purpose. 
Features designed to help shoppers find other products they might also like can be as helpful as the MC on the page. 
Ways to find other cool stuff on entertainment websites can keep users browsing happily. Helpful SC on a recipe 
webpage can make the difference between whether the recipe is a success or a failure. 

Helpful SC is content that is specifically targeted to the content and purpose of the page. For example, very helpful SC 
on a recipe page might be a feature to multiply or divide the recipe to make the right quantity of food for a given 
number of people. Very helpful SC content on a shopping page might be other popular makers or models of the same 
kind of product featured on the page. 

Large websites with many pages benefit from helpful, specialized SC. SC allows users to find related and interesting 
content on websites with many pages. Smaller websites such as websites for local businesses and community 
organizations, or personal websites and blogs, may need less SC for their purpose. A page can still receive a High or 
even Highest rating with no SC at all. 
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4.6 Functional Page Design 

High quality pages are designed to aclnieve tineir purpose: they are well organized, use space effectively, and have a 
functional overall layout. While every page is different, functional pages should have the following characteristics: 

• The MC should be prominently displayed "front and center." 

• The MC should be immediately visible when a user opens the page. 

• It should be clear what the MC actually is. The page design, organization, and use of space, as well as the 
choice of font, font size, background, etc., should make the MC very clear. 

• Ads and SC should be arranged so as not to distract from the MC — Ads and SC are there should the user 
want them, but they should be easily "ignorable" if the user is not interested. 

• It should be clear what parts of the page are Ads, either by explicit labeling or simply by page organization or 
design. 

Like everything else, functional page design depends on the purpose of the page. What constitutes functional design 
for a shopping page may be very different from what constitutes functional design for an informational page. 

Important: Some pages are "prettier" or more professional looking than others, but you should not rate based on how 
"nice" the page looks. A page can be very functional and achieve its purpose without being "pretty." Here are some 
examples of functional (but perhaps not "pretty") page design. You can click the examples to make them larger. 



Classified Advertising Website Example 


US Naval Clock Page Example 












craigslist 


SF bay area" .«= k, — 












cfHiunLm^ hau^B iabfl two 




us Naval Observatoiy Master Clock Time 










Jan. 07. 21:16:20 UTC 


Universal TijTia 










Jan. 07. 04:16:20 FM EST 


Eastern Time 










Jan. 07. 03:16:20 FM CST 


Central Time 
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Jan. 07. 02:16:20 FM MST 


Mountain Time 










Jan. 07. 01:16:20 FM PST 


Faciific Time 










Jan. 07. 12:16:20 FM AJ\ST 
Jan. 07. 11:16:20 AM 


Alaska Time 
Hawaii-Aleutian Time 






















US Nava] Ob&ei^ ator.' 

















4.7 A Satisfying Amount of Website Information 

Websites frequently include the following information: 

• About Us information. 

• Contact or Customer Service information. 

• Information about who is responsible for the content and maintenance of the website. 

Think about the purpose of the website and the type of content that might be available when considering what website 
information would be expected or demanded. High quality websites provide clear and satisfying information for their 
purpose. YMYL websites demand a high degree of trust and need satisfying website information. 

Other non-YMYL websites may need far less website information depending on their purpose. For example, an 
email address may be sufficient for some High or even Highest quality pages. 
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4.8 A Well Cared For and Maintained Website 



High quality large news websites are frequently updated, often adding news articles many times a day. High quality 
medical advice websites keep all of their informational pages current. Other websites, such as websites for small 
businesses, may be updated less frequently since addresses and store hours rarely change. 

How frequently a website should be updated depends on its purpose. However, all High quality websites are well 
cared for, maintained, and updated appropriately. 



4.9 Examples of High Quality Pages 



Webpage/Type of Content 


High Quality Characteristics/PQ Rating 


PQ Rating and Explanation 


Hiah: Government aqencv 

US Naval Observatory 
Master Clock page 


• Functional page design 

• Highly trustworthy for the purpose of the 
page 


Low^ Medi um Highest 

The purpose of this page is to display the official 
US Naval Observatory Master Clock time in 7 
different time zones. The page displays the clock 
information in a clear, easy-to-read format. The 
Naval Observatory is highly trustworthy and 
authoritative for this type of information. 


Hiqh: Humor 

Article on a humor website 


• High quality humorous IVIC 

• Positive reputation as a humor website 


Loivest Medium j-lighest 

This website is well known for its humorous, 
satirical articles. This is a cute example of a 
satisfying and funny article. 


Hiqh: Small business 1 

Local fish & chips 
restaurant 


• Authoritative for the topic of the page 


LoiVEst Medium j-lighest 

This is an "about us" page on a restaurant 
website. The purpose of the website is to give 
information about the restaurant: address, hours, 
menu, contact information, etc. This website is 
highly authoritative because it is about itself. 


Hiqh: Small business 2 
Local preservation center 


• Authoritative for the topic of the page 

• A satisfying amount of high quality MC 


LcwBst M'erfi um H1q hEst 

This is the News and Updates section of a local 
preservation center selling poultry, vegetables, 
and more. While the 2007 copyright date is 
outdated, most of the pages, including this one, 
have recent updates from 201 4. For this small 
business, the page design is functional for the 
purpose of the page. 


Hiqh: News 1 

Homepage of a newspaper 


• A satisfying amount of high quality MC 

• Positive reputation 


Lowest Medium l-ligtiest 

This newspaper has won seven Pulitzer Prize 
awards. 
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Webpage/Type of Content 


High Quality Characteristics/PQ Rating 


PQ Rating and Explanation 


Hiqh: News 2 

Article on a newspaper 
website 


• A satisfying amount of high quality MC 

• Positive reputation for the website 


LShVEst Medium Highest 

This is an article on a newspaper website which 
has won nine Pulitzer Prize awards. There is a 
satisfying amount of high quality MC. While there 
are Ads on the page, the MC is easy to find and 
the Ads are clearly labeled as Ads. 


Hiali: Bloa post 

Parenting article about 
strollers 


• A satisfying amount of high quality MC 

• Very positive reputation for the website 

• Positive reputation for this specific blog and 

a 1 ithnr 

dU Li lUI 


LShVEst Medium Highest 

This is a blog post on a newspaper which has won 
over 100 Pulitzer Prize awards. The author of this 
blog post has become known as an expert on 
parenting issues. She is a regular contributor to 
this and other media websites. 


Hiqh: ShoDDina 1 

Backpack shopping page 
on a popular store website 
(YMYL) 


• A satisfying amount of high quality MC 

• Positive reputation 

• Very helpful SC 


LOhVBSt Medium Highest 

The purpose of this page is to allow users to buy a 
school backpack. The page provides a lot of 
different backpack options, and some of them 
have user reviews. 

This is a well-known, reputable merchant, with 
detailed Customer Service information on the site. 

The SC features are particularly helpful. For 
example, the filters allow users to show results by 
categories such as color, style, and price. 


High: Shopping 2 

Bathroom decor page on a 
store website (YMYL) 


• Satisfying amount of high quality MC 

• Expertise in the type of merchandise they 
are selling 

• PnQiti\/P rpni itatinn 

• nfjoiiivc; 1 c|ju LciLiL/i 1 


LShVESt Medium Highest 

This company sells its own line of high end, 
fashionable baby and children's furniture and 
accessories. It has a positive reputation as well 
as expertise in these specific types of goods. 
Many products sold on the site are unique to this 
company. 


High: Shopping 3 

TomTom GPS page on an 
electronics website (YMYL) 


• A satisfying or comprehensive amount of 
very high quality MC 

• Expertise in the type of merchandise they 
are selling 

• Positive reputation 


LOhVBst Med i u m Hig hest 

There is a very large quantity of MC on this page. 
Note that the tabs on the page lead to even more 
information, including many customer reviews. 
The tabs should be considered part of the MC. 
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Webpage/Type of Content 


High Quality Ciiaracteristics/PQ Rating 


PQ Rating and Explanation 


Hiah: Video 1 

Saturday Night Live video 
on a video website 


• A satisfying or comprehensive amount of 
very high quality IVIC 

• Vprv hplnfiil RO 


L[jhVE5t Medium Highest 

The MC of this video page is an episode of 
Saturday Night Live, a very popular television 
show. Below the main video, there are many 
other videos that users may also be interested in. 
This SC is very helpful. 


HlQli: Video2 

"An Engineer's Guide to 
Cats" video 


• A satisfying amount of high quality IVIC 

• Everyday expertise in the topic of cat 
ownership 

• Very helpful SO 


Lowest Med i u m Higtiest 

This is a humorous, high quality, well produced 
video with more than 6.5 million views (and 
counting), illustrating the proper care and practical 
benefits of cats. The two engineers in the video 
have everyday expertise on cat ownership. For a 
humorous video like this one, expertise does not 
have to be in the form of a trained veterinarian. 

Also, this video is part of a humorous series of cat 
care videos and the SC includes many other cute 
and funny cat videos. 


Higli: Financial 

Financial Markets page on 
a newspaper website 
(YMYL) 


• High quality MC 
. Helpful SO 

• Authoritative and trustworthy 


LOhVESt Medium Higtiest 

This newspaper has won two Pulitzer Prize 
awards and has a positive reputation, though it is 
not an acknowledged expert in financial news. 


Hiah: Game 
Online game 


• A satisfying or comprehensive amount of 
very high quality MC 

• Positive reputation 


Lowest Medium Higtiest 

Highly engaging game with multiple levels that 
could entertain a child for hours. This website has 
a positive reputation for educating children about 
animals. 


High: Q&A 1 

How long do cancer 
patients live 


• Everyday expertise 

• A satisfying amount of high quality MC 


Lowest Medium HigtiEst 

The question on the page asks how long people 
live with cancer. There are many responses 
describing how long a loved one lived after 
diagnosis. There is very little medical advice and 
the focus of the page is sharing personal 
experience. Many responses are heartfelt and 
well written. 
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Webpage/Type of Content 


High Quality Characteristics/PQ Rating 


PQ Rating and Explanation 


Hiah: Q&A2 

Q&A page on which Hillary 
Clinton asks users for their 
personal opinions 


• A satisfying or comprehensive amount of 
very high quality MC 

• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 


Lmvest Medium Highest 

Since the question asks for ordinary people's 
opinions, everyday expertise is desired. Many of 
the answers are well written and thoughtful. It is 
clear that many people spent time answering this 
question, doing their best to help Hillary Clinton 
understand their oersoectives. Web research can 
help confirm that it was really Hillary Clinton 
asking the questions. 

Keep in mind that for any type of page, including 
Q&A pages, there may be a range of highest 
quality to lowest quality pages. Therefore, it's 
important to evaluate the page using the same 
criteria as all other pages, regardless of what type 
of page it is. 


Hiah: EncvcloDedia 

Encyclopedia article about 
the American Civil War 


• A satisfying or comprehensive amount of 
high quality MC 

• Pn^^itivp rpmitptinn for pmir^itp infnrmptinn 


Lowest Medium Highest 

This is a comprehensive article about the 
American Civil War. The website has a good 
reputation for accuracy. In addition to the article 
itself, there are many helpful references and 
citations to support the content. 


Hiqh: Forum 1 

Authenticating a luxury 
designer purse 


• High level of expertise for this topic; 
authoritative and trustworthy for the purpose 
of the page 

• A satisfying amount of high quality MC 


Lowest Medium Highest 

This forum is well known for discussions on luxury 
designer purses. On this particular forum page, 
members are consulting forum experts who have 
expertise authenticating bags from this brand. 
These experts can tell if a particular bag is 
authentic or fake. While there is an ad at the top 
and a few ads within the forum message, it does 
not distract from the MC, which is easy to find. 


Hiqh: Forum 2 
Landscaping an aquarium 


• High level of expertise for this topic; 
authoritative and trustworthy for the purpose 
of the page 

• A satisfying amount of high quality MC 


Lowest Med i u m Hig hest 

This discussion focuses on the landscaping for a 
particular paludarium (an aquarium with terrestrial 
and aquatic elements). There is a lot of 
discussion and interaction between forum 
members about the types of materials and 
species used in the aquarium. The posts show 
expertise in a niche topic aquarium landscaping. 
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5.0 Highest Quality Pages 



Highest pages are very satisfying pages winicin aclnieve their purpose very well. The distinction between High and 
Highest is based on the quality of MC as well as the level of E-A-T and reputation of the website. 

What makes a page Highest quality? We require at least one of the following: 

• Very high or highest quality MC, with demonstrated expertise, talent, and/or skill. 

• Very high level of expertise, authoritativeness, and trustworthiness (page and website) on the topic of the 
page. 

• Very good reputation (website or author) on the topic of the page. 

5.1 Very High Quality MC 

We will consider the MC of the page to be very high or highest quality when it is created with a high degree of time and 
effort, and in particular, expertise, talent, and skill. Very high quality MC may be created by experts, hobbyists, or even 
people with everyday expertise. Our standards depend on the purpose of the page and the type of content. The 
Highest rating may be justified for pages with a satisfying or comprehensive amount of very high quality MC. 

5.2 Very High Level of E-A-T 

Highest quality pages and websites have a very high level of expertise or are highly authoritative or highly trustworthy. 

Formal expertise is important for topics such as medical, financial, or legal advice. Expertise may be less formal for 
topics such as recipes or humor. An expert page on cooking may be a page on a professional chef's website, or it 
may be a page on the blog of a home cooking enthusiast. Please value life experience and "everyday expertise." For 
some topics, the most expert sources of information are ordinary people sharing their life experiences on personal 
blogs, forums, reviews, discussions, etc. 

Think about what expertise, authoritativeness, and trustworthiness mean for the topic of the page. Who are the 
experts? What makes a source trustworthy for the topic? What makes a website highly authoritative for the topic? 

5.3 Very Positive Reputation 

Reputation research is important when giving Highest ratings. Very positive reputation is often based on prestigious 
awards or recommendations from known experts or professional societies on the topic of the website. Wikipedia and 
other informational sources can be a good starting point for reputation research. 

For some topics, such as humor or recipes, less formal expertise is OK. For these topics, popularity, user 
engagement, and user reviews can be considered evidence of reputation. For topics which need less formal expertise, 
websites can be considered to have a positive reputation if they are highly popular and well-loved for their topic or 
content type, and are focused on helping users. 



5.4 Examples of Highest Quality Pages 



Webpage/Type of Content 


Highest Quality Characteristics of the Page 


PQ Rating and Explanation 


Highest: Technical 1 
Software tips 


• A satisfying or comprehensive amount of 
very high quality MC 

• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 


LiKvest Medium Hignest 

This page offers a satisfying amount of helpful and 
very high quality MC for the purpose of the page. 
Because the page is on the official website of the 
company that produces the software, it is highly 
authoritative. 
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Webpage/Type of Content 


Highest Quality Ciiaracteristics of the Page 


PQ Rating and Explanation 


Hiqhest: Technical 2 
Linux Kernel archives 


• A satisfying or comprehensive amount of 
very high quality MC 

• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 


1 1 ■ 1 ' 1 

Lowest Med i u m Hig hEst 

From this Wikioedia article, we learn that 
"Kernel.org is a main repository of source code for 
the Linux kernel, the base of the popular Linux 
operating system. It makes all versions of the 
source code available to all users... The main 
purpose of the site is to host a repository for Linux 
kernel developers and maintainors of Linux 
distributions." 

This website is the most authoritative source for 
information about Linux. 


Hiqhest: Financial 1 

Credit report information 
website (YIVIYL) 


• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 

• Very positive reputation 


Lowest fttedium Highest 

Users in the U.S. can obtain free credit reports on 
this website by providing their Social Security 
Number. This Wikioedia article tells us that this 
website is "the only federally mandated and 
authorized source for obtaining a free credit report." 


Hiqhest: Financial 2 
Tax forms page (YMYL) 


• A satisfying or comprehensive amount of 
very high quality MC for the purpose of the 
page 

• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 


LShVEst Medium Hignest 

The purpose of the page is to provide income tax 
forms and publications. The website represents 
an agency of the U.S. federal government that 
handles taxes, so the page and forms are highly 
authoritative and trustworthy. 


Hiqhest: Government 
aqencv 

Yosemite National Park 
information 


• A satisfying amount of very high quality 
MC for the purpose of the page 

• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 


Lowest Medium Highest 

This page has a highly satisfying amount of helpful 
information about Yosemite National Park, 
including traffic alerts and links to other parts of the 
website with additional information. Since the page 
is on the official national park website, it is highly 
authoritative. 
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Webpage/Type of Content 


Highest Quality Ciiaracteristics of the Page 


PQ Rating and Explanation 


Hiahest: Medical 1 
BMI calculator (YMYL) 


• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 


L[jiVEst M&d i u m Highest 

The purpose of this page is to provide an online 
BMI (Body Mass Index) calculator. The calculator 
is functional and easy to use. There is nice SC in 
the form of helpful articles on the left, as well as an 
app to download. 

The website represents an institute that is part of 
the National Institutes of Health. It has an 
extremely good reputation and is an expert in 
medical topics. 


Hiahest: IVIedical 2 

Page about meningitis 
(YIVIYL) 


• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 

• Very positive reputation 


Lowest Medium Hignest 

This is a meningitis reference page on a 
trustworthy and authoritative medical website. This 
website has a reputation of being one of the best 
web resources for medical information of this type. 


Hiahest: IVIedical 3 

Health information on a 
hospital page (YMYL) 


• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 

• Very positive reputation 


1 1 ' 1 ' t ■ a 

Lowest Medium Higtiest 

According to this Wikipedia article, this hospital is 
"currently regarded as one of the top 4 hospitals in 
the United States as rated bv U.S. News & World 
ReiDort." Users can trust medical information on 
this website. 


Hiahest: Medical 4 

Page about seasonal flu 
(YMYL) 


• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 

• Very positive reputation 


Lowest Med'i um Hig hest 

This is an influenza reference page on a 
trustworthy and authoritative medical website. This 
website has a reputation of being one of the best 
web resources for medical information of this type. 


Hiahest: Entertainment 
Humor website 


• A satisfying or comprehensive amount of 
very high quality MC 

• Very positive reputation 


Lowest Medium Highest 

This comedian has won multiple prestigious 
awards for his TV show. This is the homepage of 
his official website. 

Keep in mind that for any type of page, including 
humor websites, there may be a range of highest 
quality to lowest quality pages. Therefore, it's 
important to evaluate the page using the same 
criteria as all other pages, regardless of what type 
of page it is. 
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Webpage/Type of Content 


Highest Quality Ciiaracteristics of the Page 


PQ Rating and Explanation 


Hiqhest: Shoppinq 

Backpack shopping page 
(YMYL) 


• A satisfying or comprehensive amount of 
very high quality MC 

• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 

• Very positive reputation 


Lowest Me-dium Highest 

The purpose of this page is to provide information 
about, and allow users to buy, a specific type of 
school backpack. The page provides a lot of 
helpful product information, as well as 600 user 
reviews. 

Since the store produces this backpack, they are 
experts on the product, making the page on their 
own website authoritative. In addition, this store 
has a reputation for producing one of the highest 
quality and most popular school backpacks on the 
market. 


Hiqhest: Loqin 

Onlinp hankinn Innin nanp 

V^l llll Iv7 ^CLi II\II lU IW^II 1 fJCL^yj 

(YMYL) 


• A satisfying amount of high quality MC for 
its purpose 

• Positive reputation 

• Very helpful SC which contributes to a 
highly satisfying user experience 

• Highly satisfying website contact 
information 


Lowest M-ed i u m Hig hest 

This page has login functionality, as well as clear 
information about what the user is logging into. 
This is a large, popular bank with a good 
reputation. 


Hiqhest: Paqe with Error 
Messaqe 

"Custom 404" page 


• A satisfying amount of high quality MC for 
its purpose 

• Very positive reputation 

• Helpful SC which improves the user 
experience 


Lmvest Medium Highest 

This is an example of a "custom 404" page. These 
pages are designed to alert users that the URL 
they are trying to visit no longer exists. The MC of 
this page is the cartoon, the caption, and the 
search functionality, which is specific to the content 
of the website. It is clear that time, effort, and 
talent was involved in the creation of the MC. 

This publication has a very positive reputation and 
is specifically known for its cartoons. 

Keep in mind that for any type of page, including 
pages with error messages, there may be a range 
of highest quality to lowest quality pages. 
Therefore, it's important to evaluate the page using 
the same criteria as all other pages, regardless of 
what type of page it is. 


Hiqhest: Recipes 1 
Pumpkin pie recipe 


• A satisfying amount of high quality MC 

• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 

• Very helpful SC which contributes to a 
highly satisfying user experience 


Low^ Medium IHIighest 

The website is one of the most popular recipe 
websites. The page design allows users to easily 
find the recipe. There are Ads, but it does not 
interfere with the MC on the page. 

In addition, this page has a large number of user 
reviews of the recipe and other very helpful SC, 
including detailed preparation time information, a 
"recipe box" feature, etc. 
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Webpage/Type of Content 


Highest Quality Ciiaracteristics of the Page 


PQ Rating and Explanation 


Hiqhest: Recipes 2 
Basic pizza dougti recipe 


• A satisfying amount of high quality MC 

• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 

• Very helpful SC which contributes to a 
highly satisfying user experience 


LShVEst Medium Highest 

The website is one of the most popular recipe 
websites. The page design allows users to easily 
find the recipe. There are Ads, but it does not 
interfere with the MC on the page. 

In addition, this page has a large number of user 
reviews of the recipe and other very helpful SC, 
including detailed nutrition information, a "recipe 
box" feature, etc. 


Hiqliest: Recipes 3 
Clam Cliowder recipe 


• A satisfying amount of high quality MC 

• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 

• Very helpful SC which contributes to a 
highly satisfying user experience 


Lu'ivest Medi um Hig hest 

The website is one of the most popular recipe 
websites. The page design allows users to easily 
find the recipe. There are Ads, but it does not 
interfere with the MC on the page. 

This page has a very large number of user reviews 
of the recipe and other very helpful SC, including a 
shopping list, a "recipe box" feature, etc. 

Even though this user does not seem to be a 
professional chef, recipes are an example of 
everyday expertise, and this particular recipe is 
highly rated by many people. 


Hiqliest: Recipes 4 

Chocolate chip cookie 
recipe on a recipe blog 


• A satisfying or comprehensive amount of 
very high quality MC 

• Very high level of expertise, highly 
authoritative/highly trustworthy for the 

ni irnnQP nf thf^ nanp 


Lowest Medium Higtiest 

Even though this user does not seem to be a well- 
known professional chef, recipes are an example of 
everyday expertise. The author of this blog has 
documented her extensive experimentation with a 
chocolate chip cookie recipe, and her expertise is 
demonstrated in the large quantity of high or 
highest quality MC. 


Hiahest: Video 1 

The band OK Go's music 
video for the song "This 
Too Shall Pass" 


• A satisfying or comprehensive amount of 
very high quality MC 

• Very high level of expertise, highly 
authoritative/highly trustworthy 


LsiVESt Medium Higtiest 

This is a high quality, professionally produced 
video on the band's official channel. 


Hiahest: Video 2 

"Henri 2, Paw de Deux" 
video 


• A satisfying or comprehensive amount of 
very high quality MC 

• Very positive reputation for this particular 
video 


L[JhVESt Medi um Hig tiest 

This is a high quality, popular video. It won the 
"Golden Kitty Award" for "Best Cat Video On The 
Internet" at the Walker Art Center's Internet Cat 
Video Film Festival. 
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Webpage/Type of Content 


Highest Quality Ciiaracteristics of the Page 


PQ Rating and Explanation 


Hiahest: Video 3 

Saturday Night Live video 
on the TV network's 
website 


• A satisfying or comprehensive amount of 
very high quality MC 

• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 


LShVEst M Ed i u m Hig hest 

The MC of this video page is an episode of 
Saturday Night Live on the TV network's official 
website, making it highly authoritative and 
trustworthy. Below the main video, there are many 
other videos that users may be interested in. 


Hiahest: Charity 

Homepage of a charity 
(YIVIYL) 


• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 

• Very positive reputation, and the website is 
responsible for all content on the website 


Lowest Medium hlighest 

This is a highly reputable charity according to 
multiple charity rating organizations. 


Highest; Q&A 

Q&A page on abbreviations 


• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 

• Very positive reputation, and the website is 
responsible for all content on the website 


LShVESt Medium Higfiest 

The purpose of the page is to answer questions 
about the usage of abbreviations. This website has 
a very good reputation as a reference for 
information on writing, publishing, etc. It is 
considered highly authoritative and trustworthy for 
the topic of the page. 


Highest: Informational 1 

Ball gown wedding dress 
page 


• A satisfying or comprehensive amount of 
very high quality MC 

• Positive reputation 


t ■ 4 ' t • t ^ 

Lowest Medium Highest 

This page has a very satisfying amount of MC for 
users interested in ball gown wedding dresses. An 
abundance of pictures, plus options to view by 
price range, style, etc., are part of what makes this 
page so satisfying. This page is on a popular 
wedding planning website. 






Lotf^ Medium Highest 


Highest: Informational 2 
Haunted hotel in Texas 


• A satisfying or comprehensive amount of 
very high quality MC 

• High level of expertise; authoritative and 
trustworthy for the purpose of the page 


There is no formal recognition of expertise or 
authority for this topic. However, this page has 
extensive information and a very detailed account 
of how this place is haunted, including a history of 
the hotel. It's clear that time, effort, and expertise 
went into writing the content on this page. Although 
the only contact information for the website is a 
phone number and contact form, it is sufficient for 
this type of website. 
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Webpage/Type of Content 


Highest Quality Ciiaracteristics of the Page 


PQ Rating and Explanation 


Hiahest: Maaazine 

Article titled "Secret Fears 
of the Super-Rich" 


• A satisfying or comprehensive amount of 
very high quality MC 

• Very Positive reputation 


LShVEst Medium Highest 

This is a very high quality and in-depth article on an 
award winning magazine website. 


Hiqhest: Classified Ads 

Classified ads website 
homepage (YIVIYL) 


• Very positive reputation 


Low^ Medium l-lighest 

The leading website for classified ads. 


Hiqhest: Article 

Interview with musical artist 


• A satisfying or comprehensive amount of 
very high quality MC 

• Very positive reputation and the website is 
responsible for the MC on the page 

• This magazine is very well known and well 
regarded for content about artists and 
bands 


Lowest Medium l-fighest 

This magazine has a positive reputation for 
interviews with musicians. There is a large quantity 
of high quality MC. 


Hiqhest: PDF File 
Campus map 


• A satisfying or comprehensive amount of 
very high quality MC 

• High or very high level of expertise; 
authoritative and trustworthy for the 
purpose of the page 


LcwEit MEdium Higtiest 

This PDF file is a detailed campus map of a major 
univeristy, which is hosted on the official university 
website. This is a highly authoritative source for 
this information. The map includes a listing of the 
all the buildings, parking structures, parking lots, 
construction areas, etc. 



6.0 Low Quality Pages 

In this section, we will describe characteristics of Low quality pages. 

Low quality pages are unsatisfying or lacking in some element that prevents them from achieving their purpose well. 
If a page has one of the following characteristics, the Low rating is usually appropriate: 

• The quality of the MC is low. 

• There is an unsatisfying amount of MC for the purpose of the page. 

• The author of the page or website does not have enough expertise for the topic of the page and/or the website 
is not trustworthy or authoritative for the topic. In other words, the page/website is lacking E-A-T. 

• The website has a negative reputation. 

• The SC is distracting or unhelpful for the purpose of the page. 
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In addition, any of the following characteristics by itself may be a reason for a Low rating. Occasionally, these 
characteristics may be true of Medium or High quality small, amateur, or personal websites. For example, "Uncle 
Alex's Family Photos" website (a hypothetical High quality example) may have slightly odd page design or be updated 
infrequently, but the website still achieves its purpose well. Please use your judgment and think carefully about the 
purpose of the page. 

• There is an unsatisfying amount of website information. 

• The page is lacking helpful SC. 

• The page design is lacking. For example, the page layout or use of space distracts from the MC, making it 
difficult to use the MC. 

• The website is lacking maintenance and updates. 

6.1 Low Quality Main Content 

One of the most important considerations in PO rating is the quality of the MC. The quality of the MC is determined by 
how much time, effort, expertise, and talent/skill have gone into the creation of the page. 

Consider this example: Most students have to write papers for high school or college. Many students take shortcuts to 
save time and effort by doing one or more of the following: 

• Buying papers online or getting someone else to write for them. 

• Making things up. 

• Writing quickly with no drafts or editing. 

• Filling the report with large pictures or other distracting content. 

• Copying the entire report from an encyclopedia, or paraphrasing content by changing words or sentence 
structure here and there. 

• Using commonly known facts, for example, "Argentina is a country. People live in Argentina. Argentina has 
borders. Some people like Argentina." 

• Using a lot of words to communicate only basic ideas or facts, for example, "Pandas eat bamboo. Pandas eat 
a lot of bamboo. It's the best food for a Panda bear." 

Unfortunately, the content of some webpages is similarly created. We will consider content to be Low quality if it is 
created without adequate time, effort, expertise, or talent/skill. Pages with low quality MC do not achieve their purpose 
well. 

Important: Low quality MC is a sufficient reason to give a page a Low quality rating. 



6.2 Unsatisfying Amount of IVIain Content 

Some Low quality pages are unsatisfying because they have a small amount of MC for the purpose of the page. For 
example, imagine an encyclopedia article with just a few paragraphs on a very broad topic such as World War II. 

Important: An unsatisfying amount of MC is a sufficient reason to give a page a Low quality rating. 

6.3 Lacl<ing Expertise, Autlioritativeness, or Trustwortliiness (E-A-T) 

Some topics demand expertise for the content to be considered trustworthy. YMYL topics such as medical advice, 
legal advice, financial advice, etc. should come from authoritative sources in those fields. Even everyday topics, such 
as recipes and housecleaning, should come from those with experience and everyday expertise in order for the page 
to be trustworthy. 

You should consider who is responsible for the content of the website or content of the page you are evaluating. Does 
the person or organization have sufficient expertise for the topic? If expertise, authoritativeness, or trustworthiness is 
lacking, use the Low rating. 
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A warning about user-contributed content: Many websites allow almost anyone to publish pretty much anything. 
Contributors choose their own topics with no oversight. Contributors may have very poor writing skills or may have 
absolutely no expertise in the topic of the page. Contributors may be paid per article (in some cases they may be paid 
per word), and may even be eligible for bonuses based on the traffic to their pages. Depending on the topic, pages on 
these websites may not be trustworthy. 

However, some types of user-generated content have a high level of expertise. There are forums of experts on topics 
ranging from sewing to car repair to do-it-yourself home improvement projects. User-posted content can be highly 
trustworthy for many everyday topics. In fact, for some unusual hobbies, the most expert advice may exist on blogs, 
forums, and other user-generated content websites. For such topics, try to gauge the experience and expertise of the 
author(s) to decide if the page is trustworthy. Please value everyday expertise and experience for everyday topics. 

Important: Lacking appropriate E-A-T is sufficient reason to give a page a Low quality rating. 



6.4 Negative Reputation 

Reputation research is required for all PQ rating tasks unless you have previously researched the reputation of the 
website. Extremely negative, malicious, or financially fraudulent reputation information should result in a Lowest 
rating. Credible negative (though not malicious or financially fraudulent) reputation is a reason for a Low rating, 
especially for a YMYL page. 

Please exercise care when researching the reputation of businesses. Most businesses have some negative reviews, 
especially for customer service. Try to find as many reviews and ratings as possible and read the details of negative 
reviews and low ratings before inferring that the business has a negative reputation. 

Important: Negative reputation is sufficient reason to give a page a Low quality rating. Evidence of truly malicious or 
fraudulent behavior warrants the Lowest rating. 



6.5 Cliaracteristics Wliicli May Be Evidence of Low Quality 

We have very different standards for pages on large, professionally-produced business websites than we have for 
small amateur, hobbyist, or personal websites. The type of page design and level of professionalism we expect for a 
large online store is very different than what we might expect for a small local business website. 

All PQ rating should be done in the context of the purpose of the page and the type of website. 

The following sections discuss page characteristics which may be evidence of Low quality. Occasionally, these same 
characteristics may be present on smaller amateur or personal websites and are not a concern. Please use your 
judgment when deciding whether these characteristics are evidence of low quality on the page you are evaluating, or 
merely a sign of non-professional but acceptable small, amateur, or personal website design, for example, "Uncle 
Alex's Family Photos" website (a hypothetical High quality example). 



6.5.1 Unhelpful or Distracting Supplementary Content 

Some webpages are designed to encourage users to click on SC that is not helpful for the purpose of the page. This 
type of SC is often distracting or prominently placed in order to lure users to highly monetized pages. 

Sometimes, distracting and unhelpful SC is actually Ads. We can consider this both unhelpful/distracting SC as well 
as deceptive/misleading page design. 

Here is an example of a page with highly distracting SC in the right-hand column under the label "Top Posts & Pages." 

Look carefully at the SC. If it is unhelpful or distracting, think about why it is included on the page. Unhelpful or 
distracting SC that benefits the website rather than helping the user is a reason for a Low rating. 
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6.5.2 Lacking Supplementary Content 

Some pages have a small amount of SC and other pages have no SC at all. You must use your judgment when 
deciding whether such pages should receive the Low rating. 

Some "webpages" are actually PDF files or image files (such as PNG or JPEG files) which many browsers now 
display. We would not expect a PDF page or an image file to have any SO at all. 

Other types of pages may have a small amount of SC. Local libraries, small local businesses, small community 
organizations, etc. often exist to benefit the local community, and may not have a professional web designer for their 
website. A lack of SO on these types of small, local websites may not be an issue since the purpose of the page is 
often to provide information to a small community of users. 

For example. Uncle Alex created "Uncle Alex's Family Photos" website (a hypothetical High quality example) to share 
photos with family and friends. There is not a large amount of MO and not a lot of content to navigate, therefore, a 
small amount or even no SO is acceptable. 

However, we do expect websites of large companies and organizations to put a great deal of effort into creating a good 
user experience on their website, including having helpful SC. For large websites, SO may be one of the primary ways 
that users explore the website and find MO, and a lack of helpful SO on large websites with a lot of content may be a 
reason for a Low rating. 

To summarize, a lack of helpful SO may be a reason for a Low quality rating, depending on the purpose of the page 
and the type of website. We have different standards for small websites which exist to serve their communities versus 
large websites with a large volume of webpages and content. For some types of "webpages," such as PDFs and 
JPEG files, we expect no SO at all. Please use your judgment. 



6.5.3 Poor Page Design 

Sometimes, amateur websites have less professional looking page design. A page that looks like it was created in the 
1990s is OK if the page is functional for its purpose. "Uncle Alex's Family Photos" website (a hypothetical High quality 
example) may have an unusual choice of font color or background, but it still functions well for its intended audience. 

However, some pages are deliberately designed to shift the user's attention from the MC to the Ads, monetized links, 
or SC. In these cases, the MC becomes difficult to read or use, resulting in a poor user experience. These pages 
should be rated Low. 

Here are some examples of pages with poor page design, organization, layout, or use of space, which should be rated 
Low: 

• Many Ads or highly distracting Ads on the visible part of the page when it first loads in the browser (before you 
do any scrolling), making it difficult to read the MC. 

• Repeated insertion of Ads between sections of the MC, so that the page jolts the user back and forth between 
MC and Ads in a way that makes the MC difficult to read. 

• Invasive Ads, such as popups that cannot be closed. 

• A large quantity of Ads with a relatively small amount of helpful MC. 

• Text ads, placed beside or within the site's navigation links, which may confuse users. 

If a page seems poorly designed, take a good look. Ask yourself if the page was deliberately designed to draw 
attention away from the MC. If so, the Low rating is appropriate. Not sure? Keep looking for other High or Low 
quality characteristics. 
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6.5.4 Lacking Care and Maintenance 

Sometimes a website may seem a little neglected: links may be broken, images may not load, and content may feel 
stale or outdated. 

Irregular or infrequent updates may be OK for some websites; it depends on their purpose. For example, sometimes 
ordinary people set up family photo websites or personal blogs, but rarely update them. We would usually not 
consider such websites to be inadequately updated or maintained. 

Websites with medical information, legal information, tax information, etc. must be updated frequently. Users expect 
information about the most current medical thinking, current laws, this year's tax information, etc. 

Other types of websites need frequent updates as well, for example, websites with sports information, celebrity gossip, 
news, etc. Note that we are describing websites, not individual articles. Typically, news organizations publish "new" 
news articles as a way to keep their site up-to-date. Other types of websites update content on individual pages as a 
way to keep their site up-to-date, for example, encyclopedia websites. 

Important: The date on an individual news article is not a reason for a Low page quality rating. However, if a website 
is not being maintained and updated enough for its purpose, that is a reason for a Low page quality rating. 

If a website feels neglected, look carefully. Think about the purpose of the website. A year without updates for "Uncle 
Alex's Family Photos" website (a hypothetical High quality example) is fine. A few days without new articles on a 
major national news website is not acceptable. Use your judgment. If the website feels inadequately updated and 
inadequately maintained for its purpose, the Low rating is probably warranted. 



6.5.5 Unsatisfying Amount of Information about the Website 

We expect some form of website information for many or most websites. However, the amount of website information 
needed depends on the purpose of the website. For personal websites, an email address alone may be sufficient. 

Stores and websites which process financial transactions require a high level of user trust. If a store or financial 
transaction website has just an email address and physical address, it may be difficult to get help if there are issues 
with the transaction. Likewise, many other types of YMYL websites also require a high degree of user trust. Just an 
email address and a physical address may not give users enough information to feel the website can be trusted. 

Important: For YMYL pages and other pages which require a high level of user trust, an unsatisfying amount of any of 
the following is a reason to give a page a Low quality rating: customer service information, contact information, or 
information about who is responsible for the website. For other types of websites, use your judgment. 



6.6 Examples of Low Quality Pages 



Type of Webpage/Content 


Low Quality Characteristics of the Page 


Explanation 


Low: Informational 1 

40th birthday party food 
ideas 


• Low quality MC 

• Poor page design 

• The website may be abandoned 


LoWiEt Medium Highest 

This page and website have many of characteristics 
of Low quality pages. Close observation shows MC 
which contains mostly commonly known information 
and poor quality writing. The MC is broken up by 
large Ads. 
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Type of Webpage/Content 


Low Quality Characteristics of the Page 


Explanation 


Low: Informational 2 
Q&A unanswered question 


• Unsatisfying amount of MC for the 
puipuoc Ul pdyc 


Low^ Medium Highest 

Snmp wph^itpi^ rplu nn ii<?pr<5 tn prpptp virtiiallu 3II nf 
their IVIC. In this case, the IVIC is the user's 
question. If there are no answers, the amount of 
MC on the page is unsatisfying. 


Low: Article 1 


• Low or lowest quality IVIC 

• Poor page design 


— f ' t ' 1 

LryiVEit Medium High Est 

This content has many problems: poor spelling and 
grammar, complete lack of editing, inaccurate 
information. The poor quality of the MC is a reason 
for the Lowest* to Low rating. 

In addition, the popover ads (the words that are 
double underlined in blue) can make the main 
content difficult to read, resulting in a poor user 
experience. 


Article with tips for dressing 
for the office 


Low: Article 2 

Using ginger for your health 


• Low or Lowest quality MC 

• Poor page design 

• Lacking E-A-T 


Lowest Medium Highest 

This article is so poorly written that it is difficult to 
understand. There is no evidence of expertise or 
even accuracy. In addition, a large amount of SC 
and popover Ads interrupt the MC. 


Low: News 

Article about the 2010 
Forbes list of the most 
powerful women in the 
world 


• Low quality IVIC 


LShVEst Medium Highest 

Due to lack of care in the creation of this content, 
the page does not warrant a rating higher than Low. 
For example, there are multiple misspellings (Bracak 
Obama, Hilary Clinton). 


Low: Recipes 1 


• Low quality IVIC 

• Lacking SC 

• Poor page design 


Lowi^ IWediuim Highest 

This page and website have many of characteristics 
of Low quality pages. The MC is low quality and 
lacks important information. For example, it gives 
no indication of how to make a crust and doesn't list 
a pre-made crust as an ingredient. 

This page is lacking the kind of helpful SC we 
expect in recipe pages, such as user reviews or 
comments. Without such information, it's hard to tell 
if the recipe is any good. There is mild keyword 
stuffing (in pink text) at the top: "best homemade 
apple pie recipe, easy apple pie recipe, apple pie 
filling recipe, moms apple pie." This small pink text 
is not helpful for users, and is included to manipulate 
search engine ranking. 


Apple pie recipe 
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Type of Webpage/Content 


Low Quality Characteristics of the Page 


Explanation 


Low: Recipes 2 

Gluten-free New York 
cheesecake recipe 


• Very distracting and unhelpful SC 

• Poor page design 


LShVEit Medium Highest 

This is an example of poor page design. There are 
large Ads in the main column pushing down the MC. 
On the side are highly distracting SC that are 
labeled "Top Posts & Pages." It is unclear whether 
these are SC or Ads. 


Low: Q&A1 


• Lacking expertise; not very trustworthy or 
authoritative for the purpose of the page 


LowEst Medium HighEst 

There are 94 answers to this question with a few 
responses that seem helpful. Many of the posts are 
deliberately incorrect or misleading, including the top 
answer, which is labled the "best answer." 


Page about Native 
American customs 


Low: Q&A 2 

Page about a 2002 Volvo 
part 


• Misleading page design 

• Unsatisfying amount of MC for the 
purpose of the page 


LcvEst Medium Highest 

Please read the MC (areas with red boxes around 
it), including the completely unhelpful "answer." 
This answer is so unhelpful, we can consider this 
question to be unanswered. This page has an 
unsatisfying amount of MC. 

In addition to a very unhelpful "answer," the page 
design makes it difficult to distinguish the MC from 
Ads. For example, below the answer, we see a 
"sponsored answer," which has the same format as 
the real answer, but is actually an Ad and not an 
answer to the question. This page design is 
somewhat misleading or mildly deceptive. 


Low: Q&A 3 


• Misleading or potentially deceptive page 
design 

• Unsatisfying amount of MC for the 
purpose of the page 


Lowest Medium Highest 

This is a page from a Q&A site with an unanswered 
question. Q&A pages exist to answer user 
questions. Pages with unanswered questions 
should generally be rated Low because they have 
little MC (just a question and no answer) and don't 
achieve their purpose well. 

Misleading or potentially deceptive design makes it 
hard to tell that there's no answer, making this page 
a poor user experience. 

In addition to having no answer, this page has Ads 
and links to other questions (misleadingly labeled as 
"Relevant answers") displayed prominently, which 
users may mistake for answers to the question. It 
takes a moment to notice that this page actually has 
no answer. 


Page about water and soil 
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Type of Webpage/Content 


Low Quality Characteristics of the Page 


Explanation 


Low: Financial 


• Lacking expertise; not very trustworthy or 
authoritative for the purpose of the page 


Lawesi Medium Highest 

This is a page on a website which allows anyone to 
write about anything. There is no evidence that the 
author(s) have financial expertise. The article has 
grammar, spelling, capitalization, and punctuation 
errors. 


Page with advice on picl<ing 
a quality stock for 
investment (YMYL) 


Low: Medical 


• Lacking expertise; not very trustworthy or 
authoritative for the purpose of the page 


Lmvesl Medium HighKt 

There is no evidence that the author has medical 
expertise. Because this is a YMYL medical article, 
lacking expertise is a reason for a Low rating. 


Page with information about 
how long the flu lasts 
(YMYL) 


Low: Article 3 


• Low quality MC (commonly known 
information only) 

• Lacking E-A-T 

• Lacking SC 

• Poor page design 


Lowest Me-d i u m H ig h Est 

This is a YMYL topic. However, the steps listed 
here are merely commonly-known information which 
would be of little benefit to someone interested in 
adopting a child from Iraq. 

For example, step 1 says "Choose an adoption 
agency" and suggests looking in a phone book. 
There is almost no information specific to Iraq. 
There is no evidence of expertise on adoption from 
the description about the author. In addition, there 
is little helpful SC. Finally, there are a large number 
of Ads on this page, many of which interrupt the MC. 
Some Ads have the same format as links to other 
articles on this website. 


How to adopt children from 
Iraq (YMYL) 



7.0 Lowest Quality Pages 

This section describes Lowest quality pages. The examples at the end in Section 7.9 are critical for understanding 
these concepts, so please review those carefully. 

Remember that websites and pages should be created to be helpful for users. 

Important: Websites and pages which are created to harm users, deceive users, or only make money with no attempt 
to help users should be rated Lowest. 
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Here is a checklist of types of pages or websites whicin sinould always receive the Lowest rating: 



• Harmful or malicious pages or websites. 

• True lack of purpose pages or websites. 

• Deceptive pages or websites. 

• Pages or websites which are created to make money with little to no attempt to help users. 

• Pages with extremely low or lowest quality MC. 

• Pages on YMYL websites with completely inadequate or no website information. 

• Pages on abandoned, hacked, or defaced websites. 

• Pages or websites created with no expertise or pages which are highly untrustworthy, unreliable, 
unauthoritative, inaccurate, or misleading. 

• Websites which have extremely negative or malicious reputations. 

Also use the Lowest rating for violations of the Google Webmaster Guidelines . Please read these guidelines and be 
familiar with the content, particularly this section: Google Webmaster Quality Guidelines . 

Finally, Lowest* may be used both for pages with many low quality characteristics and for pages whose lack of a 
single Page Quality characteristic makes you question the true purpose of the page. 



7.1 Harmful or Malicious Pages 

Some pages are harmful or malicious. For example, pages designed to "phish" for the user's government-issued 
identification number (such as a Social Security Number), bank account information, or credit card information are 
harmful because the purpose is to steal private information. Malicious download pages are another type of harmful 
page. 

Harmful or malicious pages should be rated Lowest. 



7.2 Lack of Purpose Pages 

Sometimes it is impossible to figure out the purpose of the page despite your best effort to do so. Such pages serve 
no real purpose for users. Lack of purpose pages should be rated Lowest. 



7.3 Deceptive Pages 

We will consider a page to be "deceptive" if it is designed to deceive users or trick search engines, rather than to help 
users. The following sections in 7.3 describe several types of deceptive pages. Note that some pages may use a 
variety of these techniques. All deceptive pages should be rated Lowest. 

For more information on deceptive pages, see the Google Webmaster Quality Guidelines . 



7.3.1 Deceptive Page Purpose 

Deceptive webpages appear to have a helpful purpose (the stated purpose), but are actually created for some other 
reason. Use the Lowest rating if a webpage page is deliberately created to deceive and potentially harm users in order 
to benefit the website. 
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Here are some examples of Lowest websites or webpages with deceptive page purpose: 

• A website pretends to be or mimics tine look of a well-known store, bank, online service, or social network, but 
instead exists to "phish" or steal passwords or personal information (for example, government ID number, 
bank account information, personal medical data, etc.)- We also consider this type of website to have a 
malicious purpose. 

• A webpage claims to be a survey, perhaps offering money or a prize, but instead exists to steal passwords or 
personal information (for example, government ID number, bank account information, personal medical data, 
etc.). We also consider this type of website to have a malicious purpose. 

• A webpage claims to offer an independent review or share other information about a product, but is in fact 
created to make money for the owner of the website without attempting to help users. For example, the MC 
may contain intentionally misleading or inaccurate information created with the sole purpose of getting users to 
click on monetized links. 

• A website claims to be the personal website of a celebrity, but the website is actually created to make money 
for the owner of the website without the permission of the celebrity. For example, the page may have false 
testimonials for a product and is created for the sole purpose of getting users to click on monetized links. 

Sometimes it is difficult to determine the real purpose of a page. Look carefully and use your judgment. If you believe 
the webpage or website was created deliberately to deceive users for the benefit of the website, and with the potential 
to cause harm to the user, use the Lowest rating. 



7.3.2 Deceptive Page Design 

Some pages are designed to manipulate users into clicking on certain types of links through visual design elements, 
such as page layout, organization, link placement, font color, images, etc. We will consider these kinds of pages to 
have deceptive page design. Use the Lowest rating if the page is deliberately designed to manipulate users to click 
on Ads, monetized links, or suspect download links with little or no effort to provide helpful MC. 

Pages may use a variety of "tricks" or deceptive page design techniques. Here are two common types of pages with 
deceptive page design: 

• A fake search page is a page with a list of links that looks like a page of search results. If you click on a few of 
the links, you will see that the page is just a collection of Ads disguised as search engine results. A "search 
box" is present on the page, but if you submit a new query in the search box, you just get a different page of 
Ads disguised as search results. "Fake search" pages are examples of deceptive page design. 

• A fake directory page looks like a personally curated set of helpful links, possibly with unique descriptions. In 
reality, the links are Ads or links to other similar pages on the site. Fake directory pages are examples of 
deceptive page design. 

There are other examples of deceptive page design. For example, some pages are deliberately designed to have a 
large amount of Ads at the top so that the MC is not visible unless a user scrolls a lot to see the content at the very 
bottom of the page. In other words, some users may not even realize the MC is on the page. Another example of 
deceptive page design is to make Ads look like navigation links or SC links, or even part of the MC. 

Take a good look at the page and use your judgment. If you believe the page was deliberately created to manipulate 
users to click on Ads, monetized links, or suspect download links through deceptive page design, the page should be 
rated Lowest. 



7.3.3 Snealty Redirects 

Redirecting is the act of sending a user to a different URL than the one initially requested. There are many good 
reasons to redirect from one URL to another, for example, when a website moves to a new address. 

However, some redirects are designed to deceive search engines and users. These are a very poor user experience, 
and users may feel tricked or confused. We will call these "sneaky redirects." 

Sneaky redirects are deceptive and should be rated Lowest. 
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How to recognize sneaky redirects: 

• WInile being redirected, you notice tinat tine page redirects tinrougin several URLs before ending up on the 
landing page. 

• You notice that clicking the same URL several times takes you to different landing pages on a rotating set of 
domains. 

• You notice that you are redirected to well-known merchant websites, such as Amazon, eBay, Zappos, etc. to 
complete a transaction. 

• The URL of the landing page is different than the URL in the rating task. You should compare the two URLs to 
see if it makes sense that one would redirect to the other. A redirect from a company's old homepage to its 
new homepage on a different domain is not sneaky. Redirects from one page on one website to another page 
on the same website are also not sneaky. However, unexpected redirects from one website to a completely 
unrelated website should be considered deceptive. 

• Look at the domain registrants. If you suspect that a sneaky redirect has taken place, you should check to see 
"whois" the registrant (or owner) of the two domains. If the registrants are the same, the redirect is less likely 
to be sneaky. 



7.3.4 Using "Who Is" 

Here are instructions for checking "whois" the domain registrant: 

• Go to the site of a "whois" provider. Here are two you can use: http://www.domaintools.com and 
http://whois.mtqsv.net/default.php . Some computers also allow you to run a command like whois 
example.com from a terminal window. 

• Enter the URL of one domain in the search box on the "whois" provider's page. Sometimes, you will need to 
delete some leading or following characters. For example, if the URL is http://supportapj.dell.com/support, you 
will enter just "dell.com" in the search box of the whois provider. 

• Open another "whois" page. 

• Enter the URL of the other domain in the search box on the second "whois" page. 

• Compare the domain registrants for the two URLs. If you find that they have the same domain registrant, you 
will *typically* conclude that there is a relationship between the two websites. If they are different and do not 
seem related, it is probably a sneaky redirect and should be rated Lowest. 



7.4 Lowest Quality Main Content 

The quality of the MC is one of the most important considerations in PQ rating. In this guideline, we'll judge the quality 
of the MC by thinking about the how much time, effort, expertise, and talent/skill was involved in content creation. 

If very little or no time, effort, expertise, or talent/skill has gone into creating the MC, use the Lowest quality rating. 

All of the following should be considered either lowest quality MC or no MC: 

• No helpful MC at all or so little MC that the page effectively has no MC. 

• MC which consists almost entirely of "keyword stuffing." 

• Gibberish or meaningless MC. 

• "Auto-generated" MC, created with little time, effort, expertise, manual curation, or added value for users. 

• MC which consists almost entirely of content copied from another source with little time, effort, expertise, 
manual curation, or added value for users. 

Finally, the distinction between low and lowest quality MC is often human effort and manual curation. If you are 
struggling between "low quality MC" and "lowest quality MC," please consider how much human effort and attention 
the page has received. 

Pages with lowest quality MC should be rated Lowest. 
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7.4.1 No Main Content 

If a page is deliberately created with no MC, use the Lowest rating. Why would a page exist without MC? Pages with 
no MC are usually lack of purpose pages or deceptive pages. 

Webpages that are deliberately created with a bare minimum of MC, or with MC which is completely unhelpful for the 
purpose of the page, should be considered to have no MC 

Pages deliberately created with no MC should be rated Lowest. 



7.4.2 "Keyword Stuffed" Main Content 

Pages may be created to lure search engines and users by repeating keywords over and over again, sometimes in 
unnatural and unhelpful ways. Such pages are created using words likely to be contained in queries issued by users. 
Keyword stuffing can range from mildly annoying to users, to complete gibberish. 

Pages created with the intent of luring search engines and users, rather than providing meaningful MC to help users, 
should be rated Lowest. 



7.4.3 Gibberish or Meaningless Main Content 

Pages may be deliberately created with gibberish or meaningless (nonsense) text. 

Gibberish text may be generated by computer algorithms; for example, it may be created by copying text from many 
webpages and combining it with no attempt to produce something meaningful. Another way to create gibberish text is 
through translation tools, where text is copied from a foreign language page, run through a freely available translator, 
and then used without any human review or editing. 

No matter how they are created, gibberish pages with meaningless content are useless for users and should be rated 
Lowest quality. 



7.4.4 Automatically-Generated Main Content 

Entire websites may be created by designing a basic template from which hundreds or thousands of pages are 
created, sometimes using content from freely available sources (such as an RSS feed or API). These pages are 
created with no or very little time, effort, or expertise, and also have no editing or manual curation. 

Pages and websites made up of auto-generated content with no editing or manual curation, and no original content or 
value added for users, should be rated Lowest. 



7.4.5 Copied Main Content 

Every page needs MC. One way to create MC with no time, effort, or expertise is to copy it from another source. 

Important: We do not consider legitimately licensed or syndicated content to be "copied" (see here for more on web 
syndication). Examples of syndicated content in the U.S. include news articles by AP or Reuters. 

The word "copied" refers to the practice of "scraping" content, or copying content from other non-affiliated websites 
without adding any original content or value to users (see here for more information on copied or scraped content). 

If all or most of the MC on the page is copied, think about the purpose of the page. Why does the page exist? What 
value does the page have for users? Why should users look at the page with copied content instead of the original 
source? 
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Important: The Lowest rating is appropriate if all or almost all of the MC on the page is copied with little or no time, 
effort, expertise, manual curation, or added value for users. Such pages should be rated Lowest, even if the page 
assigns credit for the content to another source. 



7.4.6 More About Copied Content 

All of the following are considered copied content: 

• Content copied exactly from an identifiable source. Sometimes an entire page is copied, and sometimes 
just parts of the page are copied. Sometimes multiple pages are copied and then pasted together into a single 
page. Text that has been copied exactly is usually the easiest type of copied content to identify. 

• Content which is copied, but changed slightly from the original. This type of copying makes it difficult to 
find the exact matching original source. Sometimes just a few words are changed, or whole sentences are 
changed, or a "find and replace" modification is made, where one word is replaced with another throughout the 
text. These types of changes are deliberately done to make it difficult to find the original source of the content. 
We call this kind of content "copied with minimal alteration." 

• Content copied from a changing source, such as a search results page or news feed. You often will not 
be able to find an exact matching original source if it is a copy of "dynamic" content (content which changes 
frequently). However, we will still consider this to be copied content. 

Important: The Lowest rating is appropriate if all or almost all of the MC on the page is copied with little or no time, 
effort, expertise, manual curation, or added value for users. Such pages should be rated Lowest, even if the page 
assigns credit for the content to another source. 



7.4.7 How to Determine if Content is Copied 

How do you determine whether all or most of the MC is copied? How do you identify the original source of the 
content? These things can be difficult to determine, but the following steps may help. 

1 . Copy a sentence or a series of several words in the text. It may be necessary to try a few sentences or 
phrases from the page just to be sure. When deciding what sentence or phrase to copy, try to find a sentence 
or series of several words without punctuation, unusual characters, or suspicious words that may have 
replaced the original text. 

2. Search on Google by pasting the sentence or phrase (surrounded by quotation marks) inside the Google 
search box. 




%vith Ihe Wizard, he appears each time as sonmone or something* 



Here are some sentences and phrases from this page about "The Wizard of Oz" film and how to search for 
them. 
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Sentence or Phrase From a Sentence 


Query to Find the Sentence or Phrase 


A sentence on the page: 

When each traveler meets with the Wizard, 
he appears each time as someone or 
something different 


Query with quotation marks around the sentence: 

[ "When each traveler meets with the Wizard, he appears each 

time as someone or something different." ] 

Query without quotation marl<s around the sentence: 

[When each traveler meets with the Wizard, he appears each time 

as someone or something different. ] 


A phrase from the above sentence: 

with the Wizard, he appears each time as 
someone or something 


Query with quotation marl<s around the phrase: 

[ "with the Wizard, he appears each time as someone or 

something" ] 

Query without quotation marl<s around the phrase: 

[ with the Wizard, he appears each time as someone or 

something ] 


A phrase from a different sentence on the 
page: 

Winl<ies rejoice at being freed of the witch's 
tyranny 


Query with quotation marl<s around the phrase: 

[ "Winkles rejoice at being freed of the witch's tyranny" ] 

Query without quotation marks around the phrase: 

[ Winkles rejoice at being freed of the witch's tyranny ] 



3. Compare the pages you find that match the sentence or phrase. Is most of their MC the same? If so, does 
one clearly come from a highly authoritative source which is known for original content creation (newspaper, 
magazine, medical foundation, etc.)? Does one source appear to have the earliest publication date? Does 
one source seem to reasonably be the original? 

Use your best judgment. Sometimes it is clear that the content is copied from somewhere, but you cannot tell what the 
original source is. Other times the content found on the original source has changed enough that searches for 
sentences or phrases may no longer match the original source. For example, Wikipedia articles can change 
dramatically over time. Text copied from old copies may not match the current content. If you strongly suspect the 
page you are evaluating is not the original source, consider it likely to be copied. 

We have confirmed that the above text (the text in the sentence and phrases in the table) comes from an older version 
of a Wikipedia article about "The Wizard of Oz," which can be found using the Internet Archive Wayback Machine . 
Please note that searching for the above sentence and phrases will not lead you to a current Wikipedia article about 
"The Wizard of Oz," which demonstrates how confirming that the MC of a page contains copied content can 
sometimes be difficult. 

Important: The Lowest rating is appropriate if all or almost all of the MC on the page is copied with little or no time, 
effort, expertise, manual curation, or added value for users. Such pages should be rated Lowest, even if the page 
assigns credit for the content to another source. 



7.5 No Website Information 

As discussed in Section 2.5.3 , we expect most websites to have some information about who is responsible for the 
website, as well as some contact information. For websites with YMYL pages, such as online banks, we expect to find 
a lot of information about the site, including extensive customer service information. 

Think about the purpose of the website and the type of website information users would expect or demand. 

YMYL pages with absolutely no website information, or other pages where the available information is completely 
inadequate for the purpose of the website (for example, an online bank with only an email address), should be rated 
Lowest. 
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7.6 Highly Untrustworthy, Unreliable, Unauthoritative, Inaccurate, or Misleading 

Sometimes, pages just don't "feel" trustworthy. Use the Lowest rating for any of the following: 

• Pages or websites which you strongly suspect are scams (see these links for more information about Internet 
scams: Avoid Scams 1 , Avoid Scams 2 , Internet Fraud ). 

• Pages which request personal information without a legitimate reason (for example, pages which request 
name, birthdate, address, bank account, government ID number, etc.). 

• Websites which "phish" for passwords to Facebook, Gmail, or other popular online services. See here for 
information about "phishing" fraud. 

• Pages with suspicious download links, which may be malware. 

If you suspect a link is malicious, please do not click on it. You do not need to personally experience a malicious 
download in order to confirm that a website is harmful. Likewise, if you feel strongly a website is a scam, you do not 
need to engage with it to get proof. 

Pages which appear highly untrustworthy should be rated Lowest, even if you're not able to completely confirm their 
lack of trustworthiness. Please exercise caution and practice good Internet safety skills. 



7.7 Abandoned Websites or Spammed Pages on a Website 

Some websites are not maintained or cared for at all by their webmaster. These "abandoned" websites, especially 
websites which have become hacked or defaced, should be rated Lowest. 

A hacked or defaced website is a site which has been modified without permission from the website owner(s). 
Responsible webmasters should regularly check their websites for suspicious behavior and take steps to protect their 
users. Note: You will very rarely see hacked or defaced websites because most webmasters act very quickly to undo 
the damage to their website. 

However, you may encounter pages with a large amount of spammed forum discussions or spammed user comments. 
We'll consider a comment or forum discussion to be "spammed" if someone posts unrelated comments which are not 
intended to help other users, but rather to advertise a product or create a link to a website. Frequently these 
comments are posted by a "bot" rather than a real person. Spammed comments are easy to recognize. They may 
include Ads, download, or other links, or sometimes just short strings of text unrelated to the topic, such as "Good," 
"Hello," "I'm new here," "How are you today," etc. Webmasters should find and remove this content because it is a bad 
user experience. 

While a specific page on a website may have a large amount of spammed forum discussions or spammed user 
comments, it does not mean that the entire website contains only spam. 

Pages on websites which are not maintained or cared for at all should be rated Lowest. Pages with a large amount of 
distracting, unhelpful spammed user comments should be rated Lowest. 



7.8 Extremely Negative or Malicious Reputation 

Use the Lowest rating for websites with extremely negative reputations. Here are some examples originally shown in 
Section 2.7.4 . All pages on these websites should be rated Lowest because of the extremely negative or malicious 
reputation of these websites. 



Website 


Description 


Site sellina children's 
junqle aym 


Extremely negative reputation information: This business has a BBB rating of F. There is a 
news article about financial fraud. There are many reviews on websites describing how users 
sent money and did not receive anything in return. 
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Website 


Description 


Site sellina products 
related to eyewear 


Extremely negative/malicious reputation information: This website engaged in criminal 
behavior such as physically threatening users. 


Orqanization servlnq 
the hosDltalized 
veteran communitv 


Extremely negative reputation information: There are many detailed negative articles on 
news sites and charity watchdog sites describing fraud and financial mishandling about this 
organization. 



7.9 Examples of Lowest Quality Pages 



Type of Webpage/Content 


Lowest Quality Characteristics of the Page 


Explanation 


Lowest: Forum paae - 
Watch RoboCop online 


• The page has a large amount of 
spammed comments 


m- ' f ■ r ' 1 ' 1 

LiKVEst Me-dium High&st 

This is an example of a spammed forum page, 
where the goal is to try and get users to watch a 
movie online, which has nothing to do with the 
purpose of the site. 


Lowest: Copied content - 


• Lowest quality MC (copied content with 
little or no time, effort, expertise, manual 
curation, or added value for users) 


1 1 1 1 

LShVEit M E-d i u m Kig hest 

All MC is a copy of an old 

Wikipedia article. See Section 7.4.7. The current 
Wikipedia page offers much more content, links, 
references, etc. This page doesn't add much value 
for users. 


Wizard of Oz 


1 nvA/PQt" r^rinipH rrintpnt 

l_<JVvC?Ol. OUIJICU LrVJIILOllL 


• Lowest quality MC (copied content with 
little or no time, effort, expertise, manual 
curation, or added value for users) 


LcwEst Me-dium Highest 

In this article, the word "flowers" is replaced by "fs" 
throughout, which makes it difficult to find the 
original source of the article. The original source 
may be this oaqe. The modifications make it very 
difficult to read and are a poor user experience. 


"Flowers" replaced bv "Fs" 


Lowest: Unsatisfyinq 
amount of MC - Wedding 


• An unsatisfying amount of MC for the 
topic of the page 

• Poor page design 

• Lacking helpful SC for the purpose of the 
page 


jjowest Medium [Highest 

This page exhibits many Lowest query 
characteristics. 


dresses 


Lowest: Malicious, harmful, 


• Malicious, harmful, or deceptive 


L-DWEst Medium Highest 

This is a shopping checkout page, but it requests 
the user's government ID number, driver's license 
number, ATM pin number, bank account 
information, etc. 


or deceptive - Asks for 
personal information 
(YMYL) 
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Type of Webpage/Content 


Lowest Quality Characteristics of the Page 


Explanation 


Lowest: Autoaenerated 
content - About LG imorint 
lock code 


• Lowest quality MC (autogenerated 
content) 


K' 1 • -f-M 

Lowest Medium hlig^iest 

This page was created to get users to click on Ads. 
All content is copied from other sources, and the 
page is auto-generated. The fact that users are told 
that the page is auto-generated does not change the 
rating. 


Lowest: Fraudulent - 
Charity (YMYL) 


• Extremely negative or malicious 
reputation 


Lowest Medium Highest 

There are many claims of fraud and financial 
problems about this organization on reputable news 
sites and charity watchdog sites: Negative review 1 , 
Negative review 2, Negative review 3, and Negative 
review 4. 


Lowest: No website 
information - About dry 
sockets (YMYL) 


• No website information for YMYL website 

• Lacking expertise; not very trustworthy or 
authoritative for the purpose of the page 


Lowest Medium [Highest 

There is no information about who created this 
website, no contact information, and no authorship 
information. 

Medical pages require a high degree of user 
trust. Because there is no information about who 
owns this website and who created this content, we 
will consider this an untrustworthy website. 


Lowest: Fake directory 
paae - Mesothelioma 
(YMYL) 


• Deceptive page design (Ads disguised as 
MC) 

• Nn wph^iitp infnrmatinn fnr YMYI tnnir 


L[y<vest fitedi um Hig hest 

This links on the page appear to link to helpful 
articles, but in fact the links are Ads that do not go to 
articles. This is a YMYL topic, but there is no 
indication about who is responsible for this content 
or website. 


Lowest: Fake search paae 
- Chicken recipes 


• Deceptive (fake search page) 


Lowest Medium [Highest 

Advertising should never disguise itself as the MC of 
the page. Pages with Ads that are designed to look 
like MC should be considered deceptive. 


Lowest: Fake search paae 
- Chairs 


• Deceptive (fake search page) 


Lowest Medium [Highest 

This page was created to make money from clicks 
on Ads rather than to help users. 
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Type of Webpage/Content 


Lowest Quality Characteristics of the Page 


Explanation 


Lowest: Deceptive oaae 
Durpose -Viaqra 
(YMYL) 


• Lowest quality MC (copied content with 
little or no time, effort, expertise, manual 
curation, or added value for users) 

• No website information for YMYL website 


m ' \ ' \ ■ ^t— 

Lowest Medium Higfiest 

This looks like an information website for a drug. 
The real purpose of this page is to get users to this 
website, and then get them to click on the "order 
now" link, which takes users to another website. 

All MC on this page is copied with little or no time, 
effort, expertise, manual curation, or added value for 
users. Here is the oriainal source. 


Lowest: Deceotive oaae 
purpose - Walmart qift 
card survey 


• Deceptive purpose 

• Highly untrustworthy 


1 1 1 1 

LShV&it M E-d i u m High Est 

This page leads users to believe that they may 
qualify for a $1 000 Walmart Gift Card. They have to 
take a lengthy survey which is designed to get them 
to install a shopping toolbar, purchase products, and 
sign up for paid trials. The only "way out" or to 
continue after a certain point is to provide credit card 
information. Despite appearances, there is no 
affiliation with Amazon. 


Lowest: Deceptive page 
purpose and desian - 
Emerv cat toy paae 


• Deceptive page design and deceptive 
purpose 


Lowest Medium Highest 

This page claims to be an independent review of a 
product, but the only links on the page are 
prominent monetized links to purchase this product 
on another website. This page deliberately lacks SC 
so that the only "way out" is to click on a monetized 
link. This is a deceptive page design because it is 
designed to get users to click on the prominent links. 


Lowest: Deceptive paae 
purpose and desian - 
Washinq macliine reviews 


• Purpose of the page to make money with 
very little or no attempt to help users 

• Lowest quality IVIC (copied content with 
little or no time, effort, expertise, manual 
curation, or added value for users) 


Lowest Medium Highest 

This page is titled "Washing Machine Reviews," but 
there are no reviews on the page and the content is 
copied from another website. The links all go to a 
single Internet retailer selling washing machines. 
This is a deceptive page design because it is 
designed to get users to click on the prominent links. 
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Type of Webpage/Content 


Lowest Quality Characteristics of the Page 


Explanation 


Lowest: Deceptive oaae 
Durpose - Fake survey 
oaae 


• Deceptive purpose 

• Highly untrustworthy 


K' 1 ■ -t— M 

Lowest Medium IHiQ^iest 

This page contains a fake survey, which appears to 
promise users the opportunity to win an Apple 
MacBook Air, an iPhone 5c, or a Galaxy S4. 
However, after completing some survey questions 
and selecting the product the user would like to win, 
the user is redirected to website after website to 
complete more and more survey questions, while 
being asked to supply more and more personal 
information, including medical information and bank 
account information. 

There is no information about who is responsible, 
even though this website asks for highly personal 
information such as bank account numbers. Despite 
appearances, there is no affiliation with Google. 


Lowest: Deceptive oaae 
purpose - Rachael Rav 
Diet BloQ 


• Deceptive purpose and deceptive content 


Lowest Medium Highest 

The title of this page is "Rachael Ray Diet Blog," but 
the page has nothing to do with Rachael Ray or her 
diet or her products. This page exists to sell 
products using Rachael Ray's name and image. In 
fact, there is a brown-text-on-brown-background 
section at the bottom of the page (which we 
consider to be hidden text) that says "Disclaimer: 
Rachael Ray is not affiliated with nor does she 
sponsor or endorse this blog." This page is 
deceptive in spite of the disclaimer! 

This example has been annotated with red text at 
the top to point out deceptive aspects of this page. 


Lowest: Deceptive paae 
desion — Javascript alert 
box 


• Dpcpr^tivp or mi'^lpadinn nanp dp*^inn 


Lowest Medium l-lighest 

Some users might not even notice the MC because 
it is under a long list of Ads. Users may mistake the 
Ads for IVIC. 


Lowest: Keyword stuffino — 
Imodium for doas 


• Lacl<ing in purpose 

• Lowest quality IVIC (gibberish, l<eyword 
stuffing) 


Lowest Med i u m Highest 

This page has no helpful MC and no helpful 
purpose. 
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Type of Webpage/Content 


Lowest Quality Characteristics of the Page 


Explanation 


Lowest: Unmaintained 


• The website appears to be not 
maintained or updated 

• No website information for YMYL website 

• Lacking expertise; not very trustworthy or 
authoritatiye for the purpose of the page 

• Difficult to distinguish MC from SC, Ads 


1 ' 1 ■ -1— M 

LcjhVEst Medium Higfiest 

The homepage of this website indicates it was last 
updated in 2005. Additionally, usinq the Internet 
Archive Wayback Machine, we can see that this 


page hasn't changed since 2005. That is cause for 
concern on a medical website, which needs a high 
degree of trust as medical advice changes over 
time. Unmaintained medical information can be 
dangerous. 

There is no information about who created or is 
responsible for this content. There is no evidence of 
E-A-T. Also, the page design makes it very difficult 
to tell which links are SC and which are actually 
Ads. 


YMYL page - Medical 
information website (YMYL) 




Lowest: Malicious website - 


• Very negative, malicious, 
or financially fraudulent reputation 


LowBst Medium [Highest 

This website took users' money and physically 
threatened users who complained. These articles 
on Wikipedia and the New York Times describe the 
deceptive techniques used by this website and 
provide other negative information about the website 
and its owner. 


Site sellina products related 
to eyewear 


Lowest: Gibberish — 


• Lacking in purpose 

• Lowest quality MC (gibberish) 


LcwEst Med i u m Highest 

This page has no helpful MC and no helpful 
purpose. 


Celebrity site 


Lowest: Gibberish - PDF 


• Lowest quality MC (gibberish) 

• 1 ark nf niirnn*%p 


Ijiwest Med i u m Hiig hest 

This is a gibberish PDF file. We don't have any idea 
why this was created. There are no links or Ads. 
This page has no purpose and no reason to exist. 


file 


Luwcbi. onjuciibi] — nu 


• LdCKiiig III purpubc 

• Lowest quality MC (gibberish) 


Lowest MsJ i u m Hig hest 

This page has no helpful MC and no helpful 
purpose. 


treatment 


Lowest: Gibberish - NFL 


• Lacking in purpose 

• Lowest quality MC (gibberish) 


' 1 1 1 1 

Ls-vEst Medium, Highest 

This page appears to have been created just to link 
to other pages. 


ierseys 
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Type of Webpage/Content 


Lowest Quality Characteristics of the Page 


Explanation 


Lowest: Gibberish- Acai 
berry cleanse 


• Lacl<ing in purpose 

• Lowest quality IVIC (gibberish, keyword 
stuffing) 


1 ' 1 ■ -f-M 

LChVEit Medium Higfiest 

This page appears to have been created to lure 
users interested in "acai berry" to the page so they 
will click on the top links. The text is gibberish and 
full of keyword stuffing. 


Lowest! Lacl< of ouroose — 
Free credit report 


• Lowest quality IVIC (gibberish) 

• Highly untrustworthy 


Lowest Med i u mi Hlig hest 

This page has deceptive features, such as a friend 
request, a prize alert, and a download button. 


Lowest: Lacl< of ouroose - 
Las Veqas aamblina terms 


• Lacking in purpose 

• Lowest quality IVIC (gibberish, keyword 
stuffing) 


Lowest Medium Highest 

This page has no MC and no helpful purpose. 


Lowest: Hiqlilv 
untrustwortliy - Article 
about how to cure stomach 
flu 


• Highly untrustworthy, unreliable, 
unauthoritative, inaccurate or misleading 

• No website information for YMYL topic 


LcwBst Me-d i u m Highest 

This content is poorly written and uses a lot of words 
to say very little. There is no information about who 
is responsible for the content and no contact 
information for this YMYL medical topic. 


Lowest: Hiphlv 
untrustworthy - Download 
oaqe 


• Highly untrustworthy (suspect download) 


Lowest Med i u m Hlig hest 

This is a download website with only one page - this 
page. There is a lot of text (with grammar and 
spelling errors) promising that users will make 
money from this free download. The purpose of the 
page seems to be to entice users into clicking on the 
links with the promise of making money. 


Lowest: Low quality MC - 
Article about poppinq 
pimples 


• Lowest quality MC 

• Lacking expertise; not very trustworthy or 
authoritative for the purpose of the page 


LObVest Med i u m Hig hest 

This content has many problems: poor grammar, 
and sentences which are meaningless or state 
something obvious. For example: "Popping pimples 
could be or could be not the new trend of getting rid 
of them." In addition, the "About the Author" section 
of the page has an Ad, but no information about the 
author. 
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Type of Webpage/Content 


Lowest Quality Characteristics of the Page 


Explanation 


Lowest: Low quality MC - 


• Lowest quality MC 

• Lacking expertise; not very trustworthy or 
authoritative for the purpose of the page 


L-ffivEst Me-dium Highest 

The level of expertise of the author of this content is 
not clearly communicated. Providing this 
background information is particularly important for 
medical, financial, or other topics for which expertise 
is needed. This is a YMYL page. 


Article about aettinq a 
mortqaqe in Texas 


Lowest: YMYL oaqe with 
inaccurate information - 
Chest pains and smokinq 
(YMYL) 


• YMYL page with inaccurate potentially 
dangerous medical advice 


Lowest fttedium Highest 

We must evaluate this page from the point of view of 
a user visiting this page from a search engine, rather 
than a participant. The question is poorly worded 
and difficult to understand. The answers are poorly 
worded and have incorrect and potentially 
dangerous medical advice. The MC is low quality. 


Lowest: YMYL oaqe with 
bad information - Pavinq 
off loans (YMYL) 


• YMYL page with potentially damaging 
financial advice 

• Highly untrustworthy, unreliable, 
unauthoritative, inaccurate or misleading 


La-iVEst fttedium Highest 

This page gives loan advice which can be potentially 
damaging, for example, instructing people not to pay 
back their loans. The article has grammar and 
spelling errors, and the page is highly untrustworthy. 


Lowest: Potentially 


• Highly untrustworthy, unreliable, 
unauthoritative, inaccurate or misleading 

• Completely inadequate or untrustworthy 
customer service information for a 
shopping website 


Ln-wEst Med 1 u m Hig hiest 

This page is selling Nike Air Jordan shoes. When 
you look at the "Contact Us" paqe, it does not give 
the name of a company or a physical adress, which 
also cannot be found anywhere else on the website. 
This amount of contact information is not sufficient 
enough for a shopping website. 

In addition, the "Shipping and Returns" page has the 
name of another company that seems to be 
unrelated. There are also official looking logos at 
the bottom of the homepage, including the Better 
Business Bureau logo and Google Checkout logo, 
that don't appear to be affiliated with the website. 


untrustworthy shoppinq 
paqe with insufficient 
contact info - (YMYL) 



8.0 Medium Quality Pages 

In this section, we will describe pages that should get the Medium quality rating. Medium pages achieve their 
purpose. However, Medium pages lack the characteristics which would support a higher quality rating. Occasionally, 
you will find a page with a mix of high and low quality characteristics. In those cases, the best page quality rating may 
be Medium. 
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Type of Medium Page 


Discussion 


Nothing wrong, but 
nothing special 


The page achieves its purpose, however, it does not merit a High quality rating, but there is nothing to 
indicate that a Low quality rating is appropriate. 


Mixed, but with 
redeeming qualities 


The page or website has some characteristics of both High and Low quality pages, but the low quality 
characteristics are mild enough that the convincing high quality aspects make it difficult to rate the page 
Low. For example, a page with a large quantity of helpful MC, but which has somewhat poor page 
design and is lacking in SC, may be considered overall Medium quality. 



8.1 Examples of Medium Quality Pages 



Webpage/Type of Content 


Medium Quality Characteristics of the Page 


Explanation 


IVIedium: Encvclooedia 

Wikipedia article about 
baroque pearls 


• Nothing wrong, but nothing special 


Lowest Medium Highest 

This is a short Wikipedia article about baroque 
pearls, a fairly narrow topic. This page is OK for 
its purpose, but it doesn't display characteristics 
associated with a High rating. 


IVIedium: Humor 

Page from a humorous site 


• Nothing wrong, but nothing special 


LsiVEit Medium H ig ti est 

This page is from a humorous site that 
encourages users to post photos with mouths 
drawn on them. This page is OK for its purpose, 
but it doesn't display characteristics associated 
with a High rating. 


Medium: Entertainment 1 

Article about "Keeping Up 
with the Kardashians" 


• Nothing wrong, but nothing special 


Lowest Medi u m Hig hest 

This page is from a gossip website. This page is 
OK for its purpose, but it doesn't display 
characteristics associated with a High rating. 


Medium: Entertainment 2 
Article about Miley Cyrus 


• Nothing wrong, but nothing special 


Lowst Medium l-lighest 

This page is from a news/entertainment website. 
This page is OK for its purpose, but it doesn't 
display characteristics associated with a High 
rating. 
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Webpage/Type of Content 


Medium Quality Cliaracteristics of the Page 


Explanation 


Medium: Paae with Error 
IVIessaae 

"Custom 404" page 


• Nothing wrong, but nothing special 


Lowst Medium l-ligh€st 

This is an example of a "custom 404" page. 
These pages are designed to alert users that the 
URL they are trying to visit no longer exists. 
Some websites do a nice job of not only alerting 
users about a problem, but also giving them help. 

This page is on a well-known merchant website 
with a good reputation. However, this particular 
page displays the bare minimum of content 
needed to explain the problem to users, and the 
only help offered is a link to the homepage. 


IVIedium: Q&A 


• Nothing wrong, but nothing special 


Lowest Medium Highest 

This Q&A page has a discussion of different 
online merchants. There is some everyday 
expertise, but it doesn't display characteristics 
associated with a High rating. 


Q&A page wliere a user is 
lool<ing for advice on where 
to buy high-quality women's 
clothing online 


Medium: Informational 

Page about propulsion on 
the "Quality Reasoning 
Group" section of a 
university's website 


• Mixed, but with some redeeming qualities 

• Website has a good reputation 

• Content is likely to be accurate and 
trustworthy because of the website, though 
no reference links are provided 

• It's not clear who is responsible for the 
content and whether it is being maintained 


Lowest Medium l-lighest 

Although this is a well-known, highly-respected 
university with a high quality site, this page is on a 
very soecialized section of the university website. 
No author is listed and the page may have been a 
one-time project, possibly from a student, which is 
no longer maintained. 


Medium: News 

Article about a Maui 
woman's death 


• Nothing wrong, but nothing special 


\ • t ' ■ ^ 1 

Lowest Medium l-iighest 

This page is from a TV news website. This page 
is QK for its purpose, but it doesn't display 
characteristics associated with a High rating. 


Medium: Forum 1 


• Mixed, but with some redeeming qualities 




Forum page on an online 

PI ir'ti nn wphQitP A nqpt iq 

looking for help choosing a 
product category. 


Lowest Medium Highest 

The relative lack of MC is balanced a bit by the 
expertise of this forum. This forum is dedicated to 
this kind of question. 



Proprietary and Confidential - Copyright 2014 



55 



Webpage/Type of Content 


Medium Quality Characteristics of the Page 


Explanation 


Medium: Forum 2 

Forum page on a dance 
website. A user is lool^ing 
for advice on liow to wash 
ballet shoes. 


• Mixed, but with some redeeming qualities 


Lcmest Medium hlighpest 

This forum is about dance topics. Many 
participants have everyday experience washing 
ballet shoes and make recommendations based 
on their experience. 

This page is "mixed" because there is distracting 
content which makes it hard to read the MC, 
potentially because the page uses an older and/or 
outdated design. The page also has distracting 
buttons and Ads. On the other hand, there is 
some valuable everyday expertise and helpful 
MC, making Medium a good rating for this page. 


Medium: Lyrics 


• Nothing wrong, but nothing special 


Ldw^ Medium l-lighest 

There are many lyrics websites which have similar 
content. This page is OK for its purpose, but it 
doesn't display characteristics associated with a 
High rating. 


Song lyrics for the song 
"Never You/Fear Love" 


Medium: Video 

Amateur of a kitten 
meowing a lot 


• Nothing wrong, but nothing special 


Lowest Medium Highest 

This is an amateur quality video of a cute kitten 
meowing. This page is OK for its purpose, but it 
doesn't display characteristics associated with a 
High rating. 


Medium: Recipe 1 

Mexi-Chicken Casserole on 
a newspaper website 


• Nothing wrong, but nothing special 


LffiVKt Medium Highest 

This recipe was contributed by an author of 
cookbooks. However, the page has no SC related 
to the purpose of the page, such as reviews or 
links to other recipes, etc. This page is OK for its 
purpose, but it doesn't display characteristics 
associated with a High rating. 


Medium: Recipe 2 


• Nothing wrong, but nothing special 


LswEst Medium Highest 

This website is known for high quality content 
about animals and the environment. This 
particular page has a recipe for kids. There isn't 
much MC or SC. This page is OK for its purpose, 
but it doesn't display characteristics associated 
with a High rating. 


Recipe for cherry-topped 
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9.0 Page Quality Rating: Important Considerations 



At first, PQ rating may seem difficult. TInere are several aspects of the page and the website to look at and think 
about. This type of rating takes practice. Rereading sections of these guidelines and thinking about the examples 
may help when you encounter difficult rating tasks. 

Important: Do not struggle with each PQ rating. Please give your best rating and move on. If you are having trouble 
deciding between two ratings, please use the lower rating. If you are torn between three ratings, choose the one in the 
middle. 

Do not consider the country or location of the page or website for PQ rating. For example, English (US) raters should 
use the same PQ standards when rating pages from other English language websites (UK websites, Canadian 
websites, etc.) as they use when rating pages from U.S. websites. In other words, English (US) raters should not 
lower the PQ rating because the page location (UK, Canada) does not match the task location. 

These guidelines are specific to "regular" webpages. Qccasionally, you may be asked to rate a landing page which is 
not a webpage. For example, you may be asked to rate a PDF file, a PNG or JPEG image file, etc. When the landing 
page of the URL is not a webpage, some of the considerations in these guidelines may not apply. In this case, please 
use your judgment. 

Finally, this Page Quality Rating Guideline does not completely cover every aspect of page quality. If you find pages 
which you truly believe to be High or Low quality, please rate them as such, even if the reason is based on something 
not covered in this document. Please use the comment section to explain your reasoning. As always, we ask you to 
use your judgment. 



9.1 Instructions for Rating Page Quality Tasks 

The Page Quality task page is broken up into several parts: 

1 . Some initial questions about the task landing page. 

2. A "PQ grid" to record your observations about PQ characteristics of the landing page. 

3. The Qverall PQ rating slider which records your Qverall PQ rating. 

4. A comment box to explain your rating. 

Some responses to the initial questions will end the task early. If the page is Porn, Foreign Language, or Didn't Load, 
you will not fill in the PQ grid or assign an overall rating. Didn't Load should be used for pages where there is 
absolutely no content on the page created by the website. There is no MC, SC, or Ads on the page. You can see this 
Wikipedia article for descriptions of different types of error messages. 

Similarly, if you respond that the page is malicious, harmful, deceptive, or lacking in purpose, you will also not fill in the 
PQ grid or assign an overall rating. 

The PQ grid is designed to be your "note pad." It allows you to record your observations about the landing page and 
the website it belongs to. 



9.2 The Top Three PQ Considerations 

The top three most important PQ considerations are: 

• Quality and quantity of lUlain Content. Examine the MC carefully. Given the purpose of the page, evaluate 
the quality and quantity of MC. 

• Level of expertise, authoritativeness, and trustworthiness (E-A-T) of the page and the website. The 

level of E-A-T is extremely important for YMYL pages. 

• Reputation of the website. The reputation of a website is very important when the website demands a high 
level of trust. 
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These "top three" considerations will help you rate many or most pages. High or Highest quality ratings must be 
supported by evidence from at least one of these top three considerations. 



9.3 E-A-T: Page or Website? 

Several PQ characteristics are based on the landing page by itself, such as considerations about MC, SC, and page 
design, etc. Some PQ characteristics are assessed at the website level, such as contact information, reputation, and 
website maintenance. 

However, E-A-T may be based on either the page alone, or the website, or even both. 

Page level checks for E-A-T are important when a website has different authors on different pages. This is the case 
for article websites or websites like YouTube, which have user-generated content. 

Website level checks for E-A-T are important in the following situations: 

• All content on the website is produced by the same person or organization. An example is a medical website 
which is produced by a reputable physician group. 

• The content of the website is produced by different authors or organizations, but the website has very active 
editorial standards. An example of this is a science journal with very high standards for publication. 

• The website has an extremely positive reputation from experts in the topic of the website, i.e., the website is 
acknowledged to be one of the most expert, authoritative, or trustworthy sources on the topic. 



10.0 Page Quality Considerations for Specific Types of Pages 

10.1 Ratings for Encyclopedia Pages 

There are many encyclopedia-type websites. Some are highly-respected publications which are standard references. 
Some are websites with content created and edited by anonymous users with no editorial oversight or fact checking. 

In many cases, we may not know the specific author of the encyclopedia article, and must rely on results from 
reputation research of the encyclopedia when judging the expertise of an article. High and Highest quality ratings can 
only be used for encyclopedias with very good reputations for accuracy and expertise. 

As a rater, you will frequently encounter Wikipedia pages. In general, Wikipedia does have a good reputation. 
Wikipedia is a very popular resource and is generally valued for accuracy. However, there is no single author or 
organization that vouches for the accuracy of Wikipedia articles. Individual Wikipedia articles should be evaluated on 
the basis of page level checks because the quality of pages on Wikipedia varies. 

A Wikipedia article with a lot of detailed, information-rich MC, and external references can usually be rated in the High 
range. Some Wikipedia articles may even be rated as high as Highest, although this rating is usually considered too 
high for an article on a medical, financial, or legal topic, which requires an extremely high level of expertise. Wikipedia 
articles can get PQ ratings up to High if they have a lot of high quality MC, are well-researched, and do a good job 
citing their resources. Qtherwise, a PQ rating in the IVIedium range is often appropriate. Naturally, Wikipedia articles 
with very little MC should get lower PQ ratings. 



10.2 Ratings for Pages with Error Messages or No MC 

Some pages load with content created by the webmaster, but have an error message or are missing MC. 

Pages may lack MC for various reasons. Sometimes, the page is "broken" and the content does not load properly or 
at all. Sometimes, the content is no longer available and the page displays an error message with this information. 

Many websites have a few "broken" or non-functioning pages. This is normal, and those individual non-functioning or 
broken pages on an otherwise maintained site should be rated Low quality. This is true even if other pages on the 
website are overall High or Highest quality. 
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Sometimes exploring a website reveals that the individual page is not an isolated example, but rather a symptom of an 
unmaintained site (or possibly a deceptive or malicious site). When that is the case, the page should be rated Lowest 
quality. 

However, not all pages with error messages are Low or Lowest quality pages. If the purpose of the page is to 
communicate that content has been removed or is no longer available, and the page does a good job of 
communicating this message, the overall PQ rating may be higher; it may be Medium or even High. The Page Quality 
rating will depend on the website level checks and the content of the page. 



Here are some examples of "broken" or "error message" pages, as well as pages which are missing MC: 



Webpage/Type of Content 


Characteristics of the Page 


Discussion 


Deliberately created with no 
MC 


• Large amount of prominent Ads and no 
attempt to help users 


Lowest Med i u m Hig hsst 

This is an example of a page with no MC. You 
might think that the MC is "missing" due to a 
problem with this particular page, but in fact, this 
website has hundreds of pages that look the same 
way — no MC, just Ads. This website shows Ads 
with little or no attempt to help users, and should 
be rated Lowest quality. 


Paqe didn't fully load and 
has no MC 


• No MC (probably due to a temporary 
technical error) 

• Sufficient SC 

• Positive reputation 


1 ' P ' 1 • 1 ' 1 
L[}-hVKt M&dium Highest 

This page has no MC and no error message. It is 
an isolated example of a page with no MC and no 
error message on a website for a reputable 
newspaper for a town in Michigan. All of the 
navigation links work, and the page was later 
fixed. 


Error oaqe with custom 404 
messaqe 1 


• Nothing wrong, but nothing special 


Lowest Medium l-tighe^t 

This Is an example of a "custom 404" page, 
alerting users that the URL they are trying to visit 
no longer exists. Some websites do a nice job of 
alerting users about a problem and providing 
helpful tips. 

This page is on a well-known merchant website 
with a good reputation. However, this particular 
page displays the bare minimum of content 
needed to explain the problem to users, and the 
only helpful content is a link to the homepage. 


Error oaqe with custom 404 
messaqe 2 


• A satisfying amount of high quality MC for 
its purpose 

• Helpful SC which improves the user 
experience 

• Positive reputation 


Lowest Medium IHigtiest 

This is an example of a "custom 404" page, 
alerting users that the URL they are trying to visit 
no longer exists. This website does a nice job of 
explaining the issue and providing helpful tips, 
including a search box. 
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Webpage/Type of Content 


Characteristics of the Page 


Discussion 


Error paqe with custom 404 
message 3 


• A satisfying amount of high quality IVIC for 
its purpose 

• Helpful SC which improves the user 
experience 

• Very positive reputation 


L[JhVEst Medi um Hig hest 

This is an example of a "custom 404" page. 
These pages are designed to alert users that the 
URL they are trying to visit no longer exists. The 
IVIC of this page is the cartoon, the caption, and 
the search functionality, which is specific to the 
content of the website. It is clear that time, effort, 
and talent was involved in the creation of the MC. 

This publication has a very positive reputation and 
is specifically known for its cartoons, which allows 
us to go as high as High+ to Highest. 



10.3 Ratings for Forums and Q&A pages 

Ratings for forum and Q&A page can be challenging. Keep in mind the following: 

• The Main Content on forum and Q&A pages includes both the question as well as the responses and resulting 
discussions. 

• PQ ratings for a forum or Q&A page should include assessments of the expertise involved in the discussion on 
the page, as well as the reputation and expertise of the website itself. 

• Rate forum and Q&A pages from the point of view of a user who visits the page, rather than a participant 
involved in the discussion. 

Assessing expertise for discussion pages can be difficult. Forums and Q&A websites can be expert sources of 
information on a wide variety of topics. In fact, some types of information are found almost exclusively on forums and 
discussions, where community of experts can provide valuable perspectives on very specific topics. When evaluating 
a forum or Q&A website, do reputation research. Also look for evidence of an active community with a high level of 
expertise that's appropriate for the discussion topic. Participation, in-depth discussions, and new threads are signs of 
an active and engaged community. 

On forum and Q&A pages, you should accept claims of everyday expertise and experience unless you have reason to 
be suspicious of the page. Please value everyday expertise and experience, especially when there are no formal or 
authoritative sources of information. Keep in mind that everyone is an expert in his or her own life experience. Qn the 
other hand, there are many forum or Q&A pages with low quality, untrustworthy MC written by people with little 
expertise or experience. If the advice or information is not obviously based on life experience or personal expertise, 
think about how trustworthy the information is. 

For example, a patient describing his or her experience with a disease may be considered an expert in his or her own 
personal experience, but only doctors or other medical experts should offer medical advice to others. "I had the flu for 
two weeks and felt miserable" is likely an accurate report of a real life experience. However, "You should eat three 
pounds of chocolate every day that you have the flu, because it made me feel better" is untrustworthy advice. 



Proprietary and Confidential - Copyright 2014 



60 



Webpage/Type of Content 


Characteristics of the Page 


Explanation 


Q&A paae about chest 
pains and smokinq (YMYL) 


• YMYL page with inaccurate potentially 
dangerous medical advice 


Lowest Medium Highest 

We must evaluate this page from the point of view 
of a user visiting this page from a search engine, 
rather than a participant. The question is poorly 
worded and difficult to understand. The answers 
are poorly worded and have incorrect and 
potentially dangerous medical advice, making it 
lowest quality MC. 


Q&A paqe with unanswered 
question 


• Misleading page design 

• Unsatisfying amount of MC for the purpose 
of the page 


LuwEit M E-d i u m Hig hest 

In addition to having no answer, this page has 
Ads and links to other questions (misleadingly 
labeled as "Relevant answers") displayed 
prominently, which users may mistake for 
answers to the question. It takes a moment to 
notice that this page actually has no 
answer. Deceptive design and lack of an answer 
make this page a frustratingly poor user 
experience and cause this page to completely fail 
to achieve its purpose. 


Q&A paae about a 2002 
Volvo part 


• Misleading page design 

• Unsatisfying amount of MC for the purpose 
of the page 


LswEit l\*edi y m Highest 

In this example, the MC is boxed in red. Please 
read the MC, including the completely unhelpful 
"answer" to the question in the red box. This 
answer is so unhelpful, we can consider this 
question to be unanswered. This page has an 
unsatisfying amount of MC. 

In addition to a very unhelpful "answer," the page 
design makes it difficult to distinguish the MC from 
Ads. For example, below the answer, we see a 
"sponsored answer," which has the same format 
as the real answer, but is actually an Ad and not 
an answer to the question. This page design is 
somewhat misleading or mildly deceptive. 


Q&A paqe about a qame 
console issue 


• Unsatisfying amount of MC for the purpose 
of the page 


Lowest Medium Highest 

Some websites rely on users to create virtually all 
of their MC. In this case, the MC is the user's 
question. If there are no answers, the amount of 
MC on the page is unsatisfying. 
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Webpage/Type of Content 


Characteristics of the Page 


Explanation 


Q&A paae about Native 
American customs 


• Lacking expertise; not very trustworthy or 
authoritative for the purpose of the page 


Lowest M&dium Highest 

There are 94 answers to this question with a few 
responses that seem helpful. IVIany of the posts 
are wrong or misleading, including the top answer, 
which is labled the "best answer." 


Forum paqe on how to 
wash ballet shoes 


• Mixed, but with some redeeming qualities 


1 1 1 1 

LcviEit Medium High est 

This forum is about dance topics. IVIany 
participants have everyday experience washing 
ballet shoes and make recommendations based 
on their experience. 

This page is "mixed" because there is distracting 
content which makes it hard to read the IVIC, 
potentially because the page uses an older and/or 
outdated design. The page also has distracting 
buttons and Ads. On the other hand, there is 
some valuable everyday expertise and helpful 
MC, making Medium a good rating for this page. 


Q&A oaae about whether a 
Roomba will work 


• Everyday expertise for the purpose of the 
page 


Lowest Medium Highest 

Many participants share their personal 
experiences with these products, giving details 
such as how well certain models work with pet 
hair. There are many descriptions of participants' 
own experiences with this product and how well it 
works for them. 


Q&A paae about 
interviewina at Build-A-Bear 


• Everyday expertise for the purpose of the 
page 


Ill m~-^' 1 

L[jhVest Medium High Est 

The answer on this page is written by an 
employee of Build-A-Bear, so it offers a somewhat 
unique and presumably somewhat expert answer. 

The page design allows users to read the MC as 
the Ads are clearly labeled, as well as both the 
question and answer. 
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Webpage/Type of Content 


Characteristics of the Page 


Explanation 


Forum post on the maqic 
loop technique in l<nittina 


• High level of expertise; authoritative and 
trustworthy for the purpose of the page 


Lcwest Medium Highest 

The person who posted the first message on this 
forum page provides a helpful resource on how to 
master the magic loop technique in knitting. She 
also shares pictures of her own version using an 
old pair of blue jeans. With over 20 years of 
experience knitting socks, we would consider her 
to be an expert on the topic. 


Q&A page about how long 
most cancer patients live 


• Everyday expertise 

• A satisfying amount of high quality IVIC 


LowEst Medium Highest 

The question on the page asks how long people 
live with cancer. There are many responses 
describing how long a loved one lived after 
diagnosis. There is very little medical advice and 
the focus of the page is sharing personal 
experience. Many responses are heartfelt and 
well written. 


Forum paqe on 
authenticatinq a purse 


• High level of expertise for this topic; 
authoritative and trustworthy for the purpose 
of the page 

• A satisfying amount of high quality IVIC 


LShVESt Medium Highest 

This forum is well known for discussions on luxury 
designer purses. On this particular forum page, 
members are consulting forum experts who have 
expertise authenticating bags from this brand. 
These experts can tell if a particular bag is 
authentic or fake. While there is an ad at the top 
and a few ads within the forum message, it does 
not distract from the IVIC, which is easy to find. 


Forum paqe on KPIs to 
track 


• High level of expertise for this topic; 
authoritative and trustworthy for the purpose 
of the page 

• A ^ati'^fvinn amount nf hinh niiaiitv MO 


Lmvest Medium Highest 

This forum is well known for its community of 
experts on quality assurance. The repsonses 
here are authoritative and trustyworthy for the 
purpose of the page. While there are Ads on the 
page, it is easy to find the IVIC and they are clearly 
labeled as "Sponsored Link." 


Forum paqe on landscapinq 
an aquarium 


• High level of expertise for this topic; 
authoritative and trustworthy for the purpose 
of the page 

• A satisfying amount of high quality MC 


Lowest Medium Highest 

This discussion focuses on the landscaping for a 
particular paludarium (an aquarium with terrestrial 
and aquatic elements). There is a lot of 
discussion and interaction between forum 
members about the types of materials and 
species used in the aquarium. The posts show 
expertise in a niche topic aquarium landscaping. 
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Webpage/Type of Content 


Characteristics of the Page 


Explanation 


Forum paqe on cloth 
diapers 


• High level of expertise for this topic; 
authoritative and trustworthy for the purpose 
of the page 

• A satisfying amount of high quality IVIC 


LShVESt Med i u m Hig hest 

Many participants share advice and personal 
experiences on how to clean cloth diapers, 
including their own specific washing routine. This 
forum site is an excellent resource for this type of 
expertise. 


Q&A oaae where Hillary 
Clinton asl<s users for their 
personal ooinions 


• A satisfying or comprehensive amount of 
high quality MC 

• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 


L[j\VE5t Medium Highest 

Since the question asks for ordinary people's 
opinions, everyday expertise is OK. Many of the 
answers are well written and thoughtful. It is clear 
that many people spent time answering this 
question, doing their best to help Hillary Clinton 
understand their perspectives. 


Q&A paqe on abbreviations 


• Very high level of expertise, highly 
authoritative/highly trustworthy for the 
purpose of the page 

• Very positive reputation, and the website is 

'I I £ II L X Xl I'X 

responsible for all content on the website 


LShVest Medium Highest 

The purpose of the page is to answer questions 
about the usage of abbreviations. This website 
has a very good reputation as a reference for 
information on writing, publishing, etc. It is 
considered highly authoritative and trustworthy for 
the topic of the page. 



11 .0 Page Quality Rating FAQs 



Question 


Answer 


Why do we have to do all 
these steps? This takes 
a long time! 


With practice, the amount of time needed for accurate PQ ratings will decrease. The steps are 
important and are designed to help you assess many different aspects of PQ. You may be surprised 
by what you find. Pages which initially look Low quality may turn out to be Medium or High quality 
with careful inspection. The reverse may happen as well. We want your most informed, thoughtful 
opinion. 


Are we just giving High 
quality ratings to pages 
that "look" good? 


No! The goal is to do the exact opposite. These steps are designed to help you analyze the page 
without using a superficial "does it look good?" approach. 


You talked about 
expertise when rating 
MC. Does expertise 
matter for all topics? 
Aren't there some topics 
for which there are no 
experts? 


Remember that we are not just talking about formal expertise. High quality pages involve time, effort, 
expertise, and talent/skill. Sharing personal experience is a form of everyday expertise. 

Pretty much any topic has some form of expert, though there are some topics or types of pages where 
expertise is less important than other aspects for MC quality rating. 

For most page purposes and topics, you can find experts even when the field itself is niche or non- 
mainstream. For example, there are expert alternative medicine websites with leading practitioners of 
acupuncture, herbal therapies, etc. There are also pages about alternative medicine written by people 
with no expertise or experience. The MC quality ratings should distinguish between these two 
scenarios. 
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Question 


Answer 


Aren't there some types 
of pages that always 
have Low quality 
content? 


For almost any type of page, there is a range of content quality. Remember that fiigh quality content 
is defined as content that takes time, effort, expertise, and talent/skill. 

Fnr ovamnlo thoro aro Kr^th Minh anH 1 dia/ niialitv/ r'oloKritv/ nr^ccirt nanoc (^fton tho niirnnco nf thoco 
nui cAciiii)jic, Liicic die uuui niyii cti lu nj w ^uctiiLy uuit^uiiLy ^Uooi|j |Jci^co. v^iLt;ii, u lu |jui|JUoU ui li it;oC 

pages is to share scandalous, but potentially true personal information about celebrities. We can 
consider the MC of a gossip page to be high quality if it is interesting information from a somewhat 
plausible source. Gossip pages are not judged by the accuracy of their content. 


I've never seen a High 
quality page of type X. If 
there are no high quality 
pages of this type, why 
are we giving existing 
pages a Low quality 
rating? 


For some topics or types of pages, there may not be many (or any!) High quality pages now, but there 
may be in the future. We need a uniform set of standards that apply to all pages, even for pages that 
have not yet been created. 


Some of these criteria 
seem unfair. For 
example, some art pages 
do not have a purpose. 
Are these pages Low 
quality? 


Art pages do have a purpose: artistic expression. Pages created for artistic expression do not 
deserve the Low quality rating simply because they have no other purpose. Artistic expression, 
humor, entertainment, etc. are all valid page purposes. 


Are forum pages always 
Low quality? 


No. Forum pages vary. We need to evaluate forum pages using the same criteria as all other pages. 
There are some forum pages with detailed information on specific issues written by people who are 
experts in the topic being discussed. There are also shallow discussion threads with very little 
content. No type of page (shopping, news, forum, video, encyclopedia, etc.) is automatically High or 
Low quality. 


Are Q&A pages 
necessarily Low quality? 


No. Q&A pages vary. We need to evaluate Q&A pages with the same criteria as all other pages. 
Sometimes, it can be difficult to assess the accuracy of the information or the expertise/knowledge of 
the person answering the question. In these cases, you may need to do some research. If the page 
is asking for medical advice, be skeptical about the expertise of the participants in the discussion. If 
the question is asking about something related to daily life, then it is far more likely that the 
participants In the discussion have the necessary experience/expertise. 

Some Q&A pages are detailed and have accurate and reliable information. IVIany others have little 
participation or inaccurate/incomplete information. We must evaluate Q&A pages from the 
perspeciive or weu users, noi me pariicipanis in ine oiscussion. 

Remember that no type of page (shopping, news, forum, video, encyclopedia, etc.) Is automatically 
High or Low quality. 


This page has multiple 
ads on the right, in the 
middle, and at the 
bottom. Does that mai<e 
It a Low quality page? 


Not necessarily. Many High or Highest quality websites are supported by Ads. Without advertising 
and monetizatlon, some webpages could not exist because It costs money to maintain a website and 
create high quality content. The presence or absence of Ads is not by itself a reason for a High or 
Low quality rating. 

We have to look at many factors when rating Page Quality and page design is just one aspect to 
consider. Think about whether the page is functional and whether the MC is easy to find. Think about 
whether the Ads interfere with the MC or If the Ads can be mistaken for the MC. 
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Part 2: Utility Rating Guideline 

12.0 Introduction to Utility Rating 

12.1 Important Rating Definitions and Ideas 



SERP for [iphone] 



iphone 



Wab Images Maps Shupprg U^ns Mcie Searcli t^ioli 

AboLit 1,270,000,000 reaulis (0.39 seconds) 

Apple - iPhone 5 - Tlie Ihirtnest, lightest, fastest IPhone ever 

www apple camJiphoiei 

il's so thin and so ligfit yet iPhone5 tealureaa Isrgei display alastei c^ip Ihe Isl&st 
wiraless 1echrcil£>gy, an BMP iSiglit camera, and mare 
Apple Store - Apple - iPhone 5 ■ Suppon ■ Rsiiiia Display 

iPhone 5 - Buy IPhon« wth Free Sl^ipping - Apple Sto?e {U S ) 

slore apple comVj-Si'tiiH]iivse'li(ime/shop_iplione 

IPhone f The biggest iNrg to happen to iPhon» since the iPhcne Fionri5193" Also 
available unlocked and contract-'ft^a. Buy novi. 'Qjalifted cuSitiimer^s arly. 

Hrws fnr iphone 

IPhoni? Bsais Cialaxv S III But Victcrv May He Flestinn 

rmationWeek - 5 houis. ago 
irttdrmaEionWeek is Ihe leading rews and inforrnaiisn source for 
infoimalign leclinoJagy pfofessionals and business man.age^s with 
' Uchntflogy ... 

A pple feces calls lor ciiegpei iPhone 

Financial Times - by Bede McCarthy ■ 3 htjurs ago 

Annlft-J! iPhoBiB S nv^^^^ak^^s RamsLinn js t)6El-5ellir>g smaftohonp in Oi ?ni? 
ZDNet - b^' Charlifr □fibflfn& - 9 Hours ago 

||Ti£jqes for iphone - Report images 




Query: [iphone] 
Locale: English (US) 
User Location: New York, NY 



Query: This refers to the word(s), number(s), and/or symbol(s) that a user types in the search box of a search 
engine. In these guidelines, queries will have square brackets around them. If a user types the word iphone in the 
search box, we will display: [iphone]. 

User: A user is a person who types queries into a search engine when looking for information. These guidelines 
assume that the user is typing queries on a desktop or laptop computer. 

User Intent: When a user types a query, he or she is trying to accomplish something. We refer to this goal as the 
user intent. 

Locale: All queries have a locale, which is the language and location for the task. Locales are represented by a two- 
letter country code. For a current list of country codes, click here . We sometimes refer to the locale as the task 
location. 

User Location: Some queries also have a User Location, which provides more specific information about where the 
user is located. The User Location is usually a city. 

Search Engine Results Page (SERP): The page a search engine shows after a user enters a query in the search 
box. The SERP is made up of result blocks. 

Here is a screenshot of one result block from the [iphone] SERP with the parts labeled: 
Title Apple - IPhone 5 - The thinnest, lightest, fastest iPhone ever. 

URL vnwL a ppl e . c 0 m/i ph on e/ 

Snippet If s so thin and so light, yet IPhone 5 features a larger display, a faster chip, the latest 

wireless technology, an 8MP i Sight camera, and more. 

The Landing Page (LP) is the page you see after you click the title link in the result block. We will use the word 
result to refer to the result block and the landing page. 
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12.2 The Purpose of Search Quality Rating 

Your ratings will be used to evaluate search engine quality around the world. Good search engines give results that 
are helpful for users in their specific language and locale. 



12.3 Raters Must Represent the User 

It is very important for you to represent users in the locale you evaluate. You must be very familiar with the task 
language and location in order to represent the experience of users in your locale. If you do not have the knowledge to 
do this, please inform your vendor. 



12.4 Browser Requirements 

Check with your vendor for browser requirements. 

You may use helpful browser add-ons or extensions, but please do not use add-ons or extensions which interfere with 
or alter the user experience of the page. 



12.5 Ad Blocking Extensions 

Do not use add-ons or extensions that block ads for Block Utility rating or Page Quality rating. These add-ons or 
extensions may cause you to give incorrect ratings. 

As a rater, only use an ad blocking extension or add-on if specifically instructed to do so in the project-specific 
instructions. 



12.6 Internet Safety Information 

In the course of your work, you will visit many different webpages. Some of them may harm your computer unless you 
are careful. Please do not download any executables, applications, or other potentially dangerous files, or click on any 
links that you are uncomfortable with. 

It is strongly recommended that you have antivirus and antispyware protection on your computer. This 
software must be updated frequently or your computer will not be protected. There are many free and for- 
purchase antivirus and antispyware products available on the web. 

See here for a Wikipedia page on antivirus software and here for a Wikipedia page on spyware. 

We suggest that you only open files with which you are comfortable. The file formats listed below are generally 
considered safe if antivirus software is in place. 

• .txt (text file) 

• .ppt or .pptx (Microsoft PowerPoint) 

• .doc or .docx (Microsoft Word) 

• .xls or .xlsx (Microsoft Excel) 

• .pdf (PDF) files 

If you encounter a page with a warning message, such as "Warning-visiting this web site may harm your computer," or 
if your antivirus software warns you about a page, you should not try to visit the page to assign a rating. 

You may also come across pages that require RealPlayer or the Adobe Flash Player plug-in. These are safe to 
download. 
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12.7 Releasing Tasks 



Most raters have difficulty rating some tasks now and then. Some queries are about highly technical topics (e.g., 
computer science or physics) or involve very specialized areas of interest (e.g., gaming or torrents). Please release 
the task if, after research, you don't understand the query or user intent for the task or the content of the LP. 

You may release tasks for these reasons: 

• Lack expertise: You feel that you personally can't rate the query. 

• Suspicious files: The task contains unknown or suspect file formats. 

• Offensive content: You believe that the LP will be offensive to you or you feel uncomfortable visiting the LP. 

• Technical problem: The query/instructions/results have obvious rendering or formatting issues. 

• Wrong language: The query/task is in the wrong language, i.e., it cannot be understood by users in the locale 
you are rating. 



13.0 Understanding the Query 

Understanding the query is the first step in evaluating the task. 

If you don't understand the query or user intent, do web research using the Google search engine or an online 
dictionary or encyclopedia. If you still don't understand the query or user intent, please release the task. 

Important: If you research the query on Google, please do not rely on the top results on the SERP. A query may have 
other meanings not represented on Google's search results pages. Do not assign a high rating to a webpage just 
because it appears at the top of a list of search results on Google. 



13.1 Understanding User Intent 



You must understand user intent to evaluate the landing page. Think about users in your locale typing the query right 
now. What are they trying to accomplish? 



Query 


Likely User Intent 


[population of paris], English (US) 


Find the current population of Paris, France. 


[fedex], English (US) 


Navigate to the FedEx website, track a package, or find a FedEx location. 


[weather], English (US) 


Find weather information in the User Location right now. 
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13.2 Task Language and Task Location (Locale) 

All queries have a task language and task location (locale). The locale is important for understanding the query and 
user intent. Users in different parts of the world may have different expectations for the same query. 




Query: [football] Query: [football] 

Locale: English (UK) Locale: English (US) 



13.3 Queries with Multiple Meanings 

Many queries have more than one meaning. For example, the query [apple], English (US) might refer to the computer 
brand or the fruit. We will call these possible meanings query interpretations. 

Dominant Interpretation: The dominant interpretation of a query is what most users mean when they type the query. 
Not all queries have a dominant interpretation. The dominant interpretation should be clear to you, especially after 
doing a little web research. 

Common Interpretation: A common interpretation of a query is what many or some users mean when they type a 
query. A query can have multiple common interpretations. 

Minor Interpretations: Sometimes you will find less common interpretations. These are interpretations that few users 
have in mind. We will call these minor interpretations. 
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Query: [apple] 
Locale: English (US) 



Computer company 



Fruit 



Person's name 
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Query: [mercury] 
Locale: English (US) 



Planet 



Chemical element 



Car brand 



Insurance company Boat engine company 
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Minor Interpretations 
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13.4 Query Meanings Can Change Over Time 



Remember to think about the query and its current meaning as you are rating. We will assume users are looking for 
current information about a topic, the most recent product model, the most recent occurrence of a recurring event, etc., 
unless otherwise specified by the query. 






User in 2007 



iPhone 5 




III 



User in February 2013 



Query: [iplione] 
Locale; Englisin (US) 



Query: [iplione] 
Locale: English (US) 



The interpretation of the query [iphone], English (US) has changed over time as new iPhone models have been 
introduced. The first iPhone was introduced in 2007. Users searching for [iphone], English (US) at that time were 
looking for the newly introduced first iPhone model. Most users now are looking for the most recent or upcoming 
iPhone model. In the future, new models will come out and the dominant interpretation will change again. 
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13.5 Classification of User Intent: Action, Information, and Navigation: Do-Know-Go 

Sometimes it is Inelpful to classify user intent for a query in one or more of tinese tinree categories: 

• Action intent: Users want to accomplisin a goal or engage in an activity, such as download software, play a 
game online, send flowers, find entertaining videos, etc. These are Do queries: users want to do something. 

• Information intent: Users want to find information. These are Know queries: users want to know something. 

• Navigation intent: Users want to navigate to a website or webpage. These are Go queries: users want to go 
to a specific page. 

An easy way to remember this is Do-Know-Go. Classifying queries this way can help you figure out how to rate a 
webpage. Important: Many queries fit into more than one category of user intent. 



13.5.1 Action Queries: Do 



The intent of an action query is to accomplish a goal or engage in an activity on the web. The goal or activity may be 
to download, to buy, to obtain, to be entertained by, or to interact with a resource that is available on the web. Users 
want to Do something. 



Query 


Likely User Intent 


[download adobe reader], English (US) 


Download software. 


[online personality test], English (US) 


Take an online personality test. 


[what is my bmi?], English (US) 


Calculate the BMI (body mass index). 


[buy Citizen Kane DVD], English (US) 


Purchase this DVD. 


[flowers], English (US) 


Order flowers online or look at pictures of flowers. 


[bubble spinner 2], English (US) 


Play Bubble Spinner 2 online or download the game. 



13.5.2 Information Queries: Know 



An information query seeks information on a topic. Users want to Know something and the goal is to find 
information. 



Query 


Likely User Intent 


[Switzerland], English (US) 


Find travel and tourism information for planning a vacation or holiday, or 
find information about the Swiss geography, languages, economy, etc. 


[cryptology use in WWII], English (US) 


Find information about how cryptology was used in World War II. 


[how to remove candle wax from carpet], English (US) 


Find information on how to remove candle wax from carpet. 


[definition of paradox], English (US) 


Find a definition of "paradox." 


[allergies], English (US) 


Find information about allergies. 


[big dog breeds], English (US) 


Find information about big dog breeds. 



13.5.3 Navigation Queries: Go 



The intent of a navigation query is to locate a specific webpage. Users have a single webpage or website in mind. 
This single webpage is called the target of the query. Users want to Go to the target page. The most helpful page for 
a navigation query is the navigational target page. 



Query 


Likely User Intent 


[kayak], English (US) 


Go to the Kayak homepage. 


[youtube], English (US) 


Go to the YouTube homepage. 


[ebay], Italian (IT) 


Go to the Italian eBay homepage. 


[best buy store locator], English (US) 


Go to the store locator page on the Best Buy website. 


[sony customer support], English (US) 


Go to the customer support page on the Sony website. 


[canon.com camcorders], English (US) 


Go to the camcorders page on the Canon website. 
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13.5.4 Queries with Multiple User Intents (Do-Know-Go) 

Many queries have more than one likely user intent. Please use your judgment when trying to decide if one intent is 
more likely than another intent. 



Query 


Likely User Intent 


[firefox], English (US) 


Do or Go. IVIany users may want to download the Firefox browser (Do) from the official Firefox website 
(Go). 


[bmi], English (US) 


Do or Know. Some users may want to calculate (Do) their BMI (body mass index). Some users may 
want to learn what BMI is or how to interpret their BMI number (Know). 


[iphone], English (US) 


Do, Know, or Go. Users may be looking to purchase an iPhone (Do), find information (Know), or go to 
the iPhone page on the Apple website (Go). 



14.0 Utility Rating Scale for Result Blocks 

The Utility rating scale offers five rating options that apply to all types of result blocks in BU rating. 



Rating 


Description 


Vital (V) 


A special rating category for Title Link Result Blocks. 


Useful (U) 


Very helpful for many or most users. 


Relevant (R) 


Helpful for many users OR very helpful for some users. 


Slightly Relevant (SR) 


Helpful for fewer users. 


Off-Topic or Useless (OT) 


Helpful for no or very few users. 



On BU rating tasks, you will see the rating scale on the Utility Slider. 

The Utility Slider is labeled query-dependent because the utility rating is based primarily on how helpful the result (the result block 
and landing page) is for the query, i.e., the rating depends on the query. 

Please note that you may assign in-between ratings. For example, notice that there are 3 small tick marks in between R and U on 
the slider. Use in-between ratings if you think the utility of a result falls between two labels. You can either drag the slider or click on 
the point that you want the slider to land on. 

The slider uses abbreviations for the five rating options as follows: 



Utility Rating (quer^'-dependent) 
I I M I I ' I M ' I M ' I I I 

M.''A OT SR R U V 



Most result blocks have a title link and landing page. Some result blocks have no title link. 

For result blocks with a title link and landing page, the utility rating is based primarily on the MC of the LP. We call 
these blocks Title Link Result Blocks (TLRB). Please note that the title link must be at the top for the block to 
be considered a TLRB. 

For result blocks with no title link, the utility rating is based primarily on the content and prominent links inside the 
result block. We call these blocks No Title Link Result Blocks (NTRB). 
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Here are some examples of TLRBs. 



1 lkd txampies 


Description 


Kandee 

vvww.kandeej.com " 

It's time for another week of Throwtiack Tfiursday action ... and B years ago ttiis week, 1 
looked tike this at my beautiful sister, Tiffany's wedding, and here's 10 ... 


TLRB. There is a title link at the top. 




Tutorials 

for tutorials on everything from make 
up to hair to cool diy crafts and ... 










TLRB. There is a title link at the top. 


Homework Helpers: Biology - Page 281 - Google Books Result 
books. google. com/books?isbn= 1564147207 
Matthew Distefano - 2004 - Science 

D is the correct binomial nomenclature name for human beings with the proper 
notation Homo sapien. The genus name is capitalized, the species name is ... 


TLRB. There is a title link at the top. 




When Bo Bumham Insulted Justin Bleber (5 


S TeamCoco.com 


TLRB. There is a title link at the top. 


■i 


teamcoco.com > Celebs ' 
May 2, 2013 

Interview: Justin & Selena Gomez were chill 
when Bo started singing a terrible Bieber ... 


ng at Bo's show 


Images of Philadelphia eagles 

bing.com/images 
























1 




TLRB. There is a title link at the top. 


















t 

1 

f 

[ 

E 

1 


'Jews about Philadelphia Eagles 

ing.com/newB 

^^^^^^1 Philadelphia Eaqles: Which Free Aoents Should Retjm? 
^^^^^^1 Bleacher Report - 1 day ago 

^^^^^^1 Heading into the offseason, Philadelphia Eagles fans can take comfort 
Hf^^PI in knowing GM Howie Roseman has done a remar1<able fob managing the 
^^B^^H team's salary-cap... 

Philadelphia Eaqles 2014 Draft; A Scoutinq Guide for East-West Shrine Game 
Bleacher Report ■ 3 hours ago 

Vre the Phiiadelohia Eaales Poised to Be the Class of the tvlFC East? 
Sleacher Report ■ 1 day ago 

lee also: More stories ■ Top stories 


TLRB. There is a title link at the top. 
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TLRB Examples 



Description 



Images of channing latum 

bing. com/images 




TLRB. There is a title link at the top. 



See more than 59,500 images 



Videos of channing tatum 

bi ng.com/videas 




TLRB. There is a title linl< at the top. 



Jenko's Epic Splil 
YouTube 



The Eagle Official 
Trailer (2011)-C... 
YouTube 



Channing Tatum 
Worried About J.. 
YouTube 



22 Jump Street 
Trailer - Channir., 
YouTube 



We're the Millers (2013) - IMDb 
www.imdb.com/title/tt1723121/ - 

itir-k-i Rating: 7.1/10 - 143.367 votes 

Directed by Rawsor Marshall Thurber. With Jason Sudeikis, Jennifer Aniston, Emma 
Roberts, Ed Helms, A veteran pot dealer creates a fake family as part of his ... 
Full Cast & Crew - We're the Millers (201 3) Poster ■ Parents Guide ■ Trivia 



TLRB. There is a title linl< at the top. 



We're ttie Millere - Official Trailer [HD1 - YouTube 

ww/w. youtube.com/watch?v=0Vsy5KzsieQ-' 
May 22, 2013 - Uploaded by Warner Bros, Pictures 
https ://www.fa ceboo k.co m/wereth em il le rs 
http://www.werethem(llers.com In theaters August 7th. Watch the 
Red ... 



TLRB. There is a title link at the top. 



Courtyard San Francisco Airport/Oyster Point Waterfront - IWaniott 
www ,marriott,com/.,./sfoop-courtyard-san-francisco-airport-oyster-point,,, ^ 
With a reinvented lobby and bistro, the waterfront Courtyard South San Francisco 
Airport hotel has redefined the travel experience to provide guests with more... 
3.6 * A ■* 12 Google reviews ■ Write a review - $1 99T 

(S) 1300 Veterans Blvd. South Sar Francisco, CA 94080 

(650)871-4100 
Maps & Transportation - View photos 



TLRB. There is a title link at the top. 
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TLRB Examples 



Description 



Map of 121 Fulton St, Mew York. MY 10038. United States 

bing. com/maps 

View large map ■ Drive fram ■ Drive to 

■ ' ■ Ctty Hill 

Park a 




TLRB. There is a title link at the top. 



Movies for all matching theaters 
Rids Along 

Jack Ryan: Shadow Recruit 
Lone Survivor 
Devil's Due 

TtveNutJob 

Frozen 

+ Show nnore movies 



1hr40min PG-13 Corned/ trall«i- | 

1hr46min PG-13 Action Tra H<ir~| 

2hr1nnm R Action | Tullw | 

1hr29min R Honor TraFlar I 

1hrZ6min PC Animation T raHiTI 

1hr48min PG Animation Trallw I 



TLRB. There is a title link at the top. 



II Capuccino Ristorante - Sag Harbor, NY - Italian Restaurant ... 
htt ps ://www.f acebook .com/pages/ll-Capuccino,.,/190274376291 - 

Rating: 4,7 - 115 votes 
II CHpuccino Ristorante, Sag Harbnr, NY. 1 08S likes - 22 talking about ttiis 1427 
were here. II Capucclno*s is more Itian just s cozy italian restaurant it Is a place ... 



TLRB. There is a title link at the top. 



Sctiolarlv articles for survey of enacted curriculum 

,., To Describe the Enacted CurrleiilLim--The Reform Lip ,,. - Smitfispn - Cited by 64 
,., :2000 Report. Using the Survey of Enacted Curriculum ,., - Kim - Cited by 9 
■ ■■ surveys of enacted curriculum to advance evaluation ... - Blanit- Cited by 23 



TLRB. There is a title link at the top. 



V: Visa Inc (New York Stock Exchange) 

MSN Money ■ Data in USD ■ Jan 17, 05:58 PM EST ■ 20 min delayed 

232.18 A.I 0.41 (4.69%) 

232,00 -0.18 (-0.08%) After Hours 




Janl7 11:30 

By Morningstar 



Open; 222.05 
Vol: 9.59 M 
P/E: 30,57 



Day's L/H: 222,04 -232.26 
Avg Vol: 2.87 IVI 
Mkt Cap: 180,57 B 



TLRB. There is a title link at the top. 
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TLRB Examples 



Description 



Images for purple flower - Report images 




TLRB. There is a title linl< at tlie top. 



Track USPS package 9400 1096 9993 8733 9325 96 
www.usps.com 



TLRB. There is a title link at the top. 



Here are some examples of NTRBs. 



NTRB Examples 


Description 






Apple 0... 






L- 1 ■ u 1 r 1.; r el ectror i cs CO m p a r y ^ 






Apple Inc.ls an American multinational 1 ^^^^^^^fl 
corporation headquartsred in Cupertino, a^^^^^^^^^. 
California, that designs, develops, and sells ^HB^^H^^I^B 

consumer electronics, computer software and ^ W 
personal computers. . . -^..^ 






Stock price: AAPL (NASDAQ) $540.67 -13.58 (-2.45%) 

Jan 17. 4:00 PM EST ■ DisdairTDr 






Founded: April 1, 1976, Cupertinn, CA 






CEO: Timothy D. Cook 
Headquarters: Cupertino, CA 




NTRB. There is no title link in this block. 


Founders: Steve Jobs, Ronald Wayne, Steve Wozniak 






People also search for 

^^^n ^ /d0Ll\ P^i""'" Sprint'> 
^^^^ at&t V J 

Best Buy AT&T Dell Microsoft Sprint 

Mobility Corporation Corporation 

1 






Feedback t More Info 
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NTRB Examples 



Description 
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NTRB. There is no title link in this block. 




Scientific name 
Speed 

Mass 



Gray wolf 

Canis tupus 

50 -60 km/h (Running) 
8-9 km/h {Loping pace) 

36 - 38.5 kg (Female, Adutt) 
0.3-0.5 kg (Newborn) 
43 - 4S kg (Male, Adult) 




49 - 65 km/h (Adull, In Siiort 

0.7 - 1 .4 1^ (Newborn) 
65 -167 Kg (Female, Adult) 
90-306 kg (Male. Mult) 



F«e(lt7BCk / Mere Info 



NTRB. There is no title link in tliis blocl<. 



1 30 Australian Dollar equals 

114.00 US Dollar 



r 



130 Australian Dollai^ 



114.00 USI^allar 



1,2 
1.0 
0.8 
0.6 




2010 2011 E012 2013 2014 



NTRB. There is no title link in this block. 



See results about 

Toys "R" Us 
Retail company 

Toys "R" Us, Inc. is a dedicated toy and 
juvenile-products retailer kiunded in 1948 and . 



NTRB. There is a blue link, but it is not at the 
top, so we consider this a NTRB. 



Feedback / More Info 
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NTRB Examples 



Description 



® 

world Ttflde 



NEW 

chCTiwtsia YO.RK center 



□ 



oaiyHall 



/?> Wwld Trjd( 

Cenlei 

' --'Memorial 

cortlmdisia fulttmSi 
o 



n Chambers' Si 




IT I f Bro«d St □ 

121 Fulton St New York, NY 10038 
Transit Fuiton SI 



€2014 Google. Sanborn 
Getdirsclions 



NTRB. There is no title link in this block. 



Rise of tine Guardians, Reviews 

2012 Film 



★★★i 7.3/10 

IIWDb - 74,461 votes 



57% 



Metacritlc - 34 reviews 



F9>Qi:lbai2l< / Mors Info 



NTRB. There is no title link in this block. 



n 



Need help? In the U.S., call 1-800-273-8255 
National Suicide Prevention Lifeline 



NTRB. There is no title link in this block. 



What are the symptoms of west nile? 

About so out of 100 people who have West tvJile have no symptoms.2 When 
symptoms do appear, they begin 3 to 14 days after the mosqjito bite.l^ild symptoms 
include: ... 

read full answer on webmd.com 

... Mild disease, generally called W^st Nile fever, may cause some or all of the 
following symptoms; 

■ Abdominal pain 

■ Diarrhea ... 

read full answer on njh.gov 



NTRB. There is no title link in this block. 
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NTRB Examples 



Description 



Visa Inc 

NYSE: V- Jan 17 S:38 PM ET 

232.18 +10.41 (4.69%) 

After Hours: 232 .OB -0.10 {-0.04%) 
232 



228 

224 J 










1 12 1 


2 3 4pin 


Id Ell 


1m 6in 


1y Sy max ^ 



Open 222.05 

Hiah 232.26 

Low 222.04 

Volume 9,586.108 

AvgVol 3,031.000 

MktCap 147.79B 



NTRB. There is no title link in tliis block. 



Google Finance - Yahoo Fmanee ■ MSN Mone^ 



Alice, TX 

Friday 5:00 PM 
Clear 



66 



I "C Precipitation: 10% 

Humidity: 20% 
Wind: 15 mph 

Tamparatur* Praclplladon 




NTRB. Tliere is no title link in this block. 



5PW BPM 11PM 

Sat Sun 



2AM SAM em 11AM 2PM 

Mon Tue Wad Thu Frt 



68" 50= 76- 52" 



Tie Weawier Channel ■ Weather Underground . AccuWeather 



Send feedback 



extrusive rock 



Web definitions 

Extrusive refers to the mode of igneous volcanic rock fonnatlon in which hot magma 
frorr inside the Earth flows out onto the sur^ce as lava or explodes violentiy into the 
atrrosphere Is fall back as pyrodasticsorluff. ... 

http://en.wikipedia.orgftvi ki/Extru s ive_Ro ck 



NTRB. There is no title link in this block. 



The stem is the rear or aft-most pari of a ship or boat, technically 
defined as tine area built up over the sternpost, extending upwards from 
the counter rail to the taffrail. The stem lies opposite of the bow, the 
foremost part of a ship. 

Stern - Wikipedia. the free encvciooedia 
€n. w kipedia .or^wikiystem 



NTRB. This one is tricky because there is a 
prominent blue link, but it is not at the top. 
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NTRB Examples 


Description 






The daughter of Jedi Knight i^akin Skywalker and Senator Padme 
Amidala of Naboo, Leia was the younger twin sister of Luke Skywalker, 
and, stiortly after her birlti, she became the adopted daughter of Bail 
Organa and Queen Breha of Alderaan, making her a Princess of 
Alderaan. 


1 


NTRB. This one is tricl<y because there is a 
prominent blue linl<, but it is not at the top. 


Leia Oraana Solo - Wookieeoedia. the Star Wars Wiki 
starwars .wi kia. comfwiki/Leia_Orgarja_Solo 










Sherwin-Williams Fioorooverina ia 48iOEari¥Rii 
www.5herwin-williams.com V Mt Crawford. VA 
Googie+page (540)432-0869 






Sherwin-Williams Paint Store fe 1820 Evelyn Byrd Ave 
www.sherwin-williams.corr #120 
Google+ page Harlsorburg, VA 

(540)434-4340 

Sherwin-Williams Paint Store (f. 231 9 w Main st 
www.5herwin-williams.com ' Waynesboro, VA 
Googie+page (540)942-5616 


NTRB. This one is tricky because there are 
several blue links. However, since there is no 
main link at the top that applies to all of the 
content in the block, this is a NTRB. 


See results for sherwin Williams bridgewater va on a map » 






Jon A. Husted 






Ohio. Secretary of State ^^HHl^^l 

IS 




NTRB. There is no title link in this block. 


FipBdtiBiCk / Mar^ nfa 






1 

1 1 :36 AM 

Tuesday, January 21. 2014 (HST) 
Time in Hawaii 

1 


NTRB. There is no title link in this block. 








Monday, May 26 

Memorial Day 2014 




KITRR Xhoro ic nn titio link in thic Klnr'l^ 


FBBcSbadt 
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NTRB Examples 



Description 



American Airlines Flight 34 

Landed 



LAX 



>)-JFK 



Departed Los Angeles, today 
Scheduled B:00 AM Terminal Gate 

7:56 AM 4 46A 



Arrived New York, today 

SchetJy led 4:25 PM Terraina! Gate 

4:22 PM 8 C33 



NTRB. There is no title link in this block. 



Miami Dolphins 

S-S , 3 rd in AFC Easte m Divisici ri 




Scores & Schedule 




Sun^Dec15 vs. Patriots 


W20 -24 Final 


Sun, Dec 22 @ Qills 


LO-19 Final 


Sun, Dec 29 vs. Jels 


1. 20 ■ 7 Final 


Al\ trm&s are m Easfem 77me 





NTRB. There is no title link in this block. 



reciclar 

"recycle" In Spanish 



QooglB Trnnal9l9 



NTRB. There is no title link in this block. 



movies 




I^Sjb^ 1 



NTRB. There is no title link in this block. 







Length 








' 164 


= ' 6.45669 




Millimetei' t 


Inch 4 









NTRB. There is no title link in this block. 
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NTRB Examples 


Description 






KQtels near Franklin. TN - 




Cwl$()ring3 NBStnrtElQ - Sm-.. &»I Springs $v C»ns. . . HaslnrinaA^ren. . . Su«lBS NatfivBI... Su>}es Natlwill. . . ^^^^^^^^^^H 


NTRB. There is no title link in this block. 






Sometimes it is really hard to tell whether a block is a TLRB or whether it is a NTRB. Remember, you can fall back on 
this rule: a TLRB must have a title link at the top of the block. 


Other examples of TLRB and NTRB blocks 


Explanation 


Leia Orqana Solo - Wookieepedia, the Star Wars Wiki 
starwars.wikia.com/wiki/Le ia_Orgara_S olo • 

Her adoptive parenis could not have children of their own, so Leia became the crown 
princess, and» as such, was formally relerred lo as Ihe Princess Leia ... 
Saba Sebatyne - Bail Prestor Organa - Slave Leia costume - Breha Organa 


TLRB. This block has a title link at the top. 
Your rating should be based primarily on the 
landing page. 


The daughter of Jedi Knight Anakin Skywalker and Senator Pad me | 
Amidala of Naboo, Leia was the younger twin sister of Luke Skywalker, 
and, shortly after her birth, she became the adopted daughter of Bail 
Organa and Queen Breha of Alderaan, making her a Princess of 
Alderaan. 

Leia Oraana Solo - Wookieepedia, the Star Wars Wiki 
starwars .wi kia. com/wi ki/Leia_Organa_Sol o 


IN 1 FiD. 1 lllo UIUOIa UUco IIUL I IciVt; Ci LILIo lllll\ dL 

the top. Your rating should be based primarily 
on the content in the block. 






Map of 121 Fulton St. New York. NY 10038. United States 

bing. corny maps 

Vi€tw large map ■ Drive from ■ Drive lo 

/ Financial .^^M 

&?-. // , „, % * . »»'iSi,';ir'di>^ri;rK.rj»r''^£0i3 Nokia 


TLRB. This block has a title link at the top. 
Your rating should be based primarily on the 
landing page. 
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Other examples of TLRB and NTRB blocks 



Explanation 



® 



World Ttwte 
Cenler 



NEW 

ChmbttsSlO YORK cVri'/fP. 



□ 

^'-^ Memorwl 



Pork PI 



Center 



Patf 



cortlSdlsia FullonSl % # 1^, 



B RIDGE 




Titanic 




121 Fultor St New York, NY 10038 
Transit Fuflan SI 



^jjjilala 92014 &iiigt>. Swtxm 
G«t dir«c4ions 



NTRB. This block does not have a title link at 
the top. Your rating should be based primarily 
on the content in the block. 



V: Visa Inc (New York Stock Exchange) 

MSN Money ■ Data in USD ■ Jan 17, 05:58 PM EST ■ 20 min deiayed 
232.18* +10.41 (4.69%) 

232.00 -0.18 (-0.08%) After Hour^ 
Open: 222.05 Day's UH: 222.04 - 232.26 
Vol: 9.59 M AvgVol: 2.87 M 
P/E: 30.57 



Jaai? 11:36 1:30 3:3C 

By MoTHTigstar 



Mist Cap: 180.57 B 



TLRB. This block has a title link at the top. 
Your rating should be based primarily on the 
landing page. 



Visa Inc 

NYSE: V - Jan 1 7 5:30 FWI ET 

232.18 +10.41 (4.69%) 

After Houre: 232.09 -0.10 (-0.04%) 




4pm 



Open 

High 

Ujw 

Volume 

Avg Vol 

MktCap 



Z22.0S 

232.26 

222.04 

9,S«6.10B 

3.031.000 

14 7.79 B 



NTRB. This block does not have a title link at 
the top. Your rating should be based primarily 
on the content in the block. 



GoogJB FJjiarvoe . vadoo Flflancs - MSN Moray 



Rating No Title Linic Result Blocks (NTRB) 



TLRBs should be ratecJ based primarily on the MC of the landing page. This means the content of the block itself is 
not an important consideration in the rating. 

On the other hand, NTRBs are often special informational result blocks designed to give users information directly on 
the results page. In some projects, you may see blocks referred to as "Special Informational Result Blocks" or 
"SIRBs." NTRBs (and SIRBs) should be rated based primarily on the content in the block itself. 

There are some special challenges with rating NTRBs. Here are three important considerations: 

1 . All result blocks (TLRBs and NTRBs) are presented as "screenshots" or images of search results with 

prominent links enabled. This works well for TLRBs, but NTRBs may have interactive features on the search 
results page. Unfortunately, a screenshot or image of a NTRB will not function as it would for a real user. For 
the purpose of rating, please assume that the NTRB does function, even though you are unable to use the 
buttons, tabs, or other features in your rating task. 
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Examples of Interactive NTRBs 



Explanation 



2.25 





flBd 




x! 


[ 


) 


% 


AC 


im 


sin 


In 


7 


B 


9 




n 


cos 


log 


4 


5 


6 


X 


a 


tan 


V 


1 


2 


3 




Ans 


EXP 




0 






+ 



NTRB. This is a "screenshot" or image of an 
interactive calculator result. The buttons do 
not function in the screenshot. When rating, 
assume that users are able to use the 
interactive calculator features. 



Visa !nc 

NYSE: V - Jan 17 5:38 PM ET 

232.18 +10.41 (4.69%) 

Aflar Hours; 232.08 -0.10 (-0.04%) 



232 VWXw^ 


22a -J. 

224 ^ 




















IQam 11 


12 1 


I 


3 




Id Sd 1r 


n Bm 


IV 


«y 


IIBX 



Open 
High 
Law 



222 .OS 
232,26 
222.0J 



VnluiTH 9,Sa6,10 



MktCap 147.79B 



Google f nance - Va^oo Finance ■ MSM MQnay 



NTRB. This is a "screenshot" or image of an 
interactive stock price result. The buttons do 
not function in the screenshot. When rating, 
assume that users are able to view the prices 
for each of the "buttons": 1 day, 5 days, 1 
month, 6 months, 1 year, 5 years, etc. 



Alice, TX 

Friday 5:00 PM 
Clear 



66 



■F I "C 



Precipitation: 10% 
Humidity: 20% 
Wind: iSmph 



TDitipBrAhirft PreclpllatlDn 



4a 



41 



5PM 

Fri 



BPM 11 PM 2AM SAM BAM 11AM ZPM 

Sat Sun Mon Tue Wet) Ttiu Fri 



70' 41- 73' 50" 76* 50" 82' 60' 63° 't 3' 66" 48" 68' SO" 75' 52" 



Tlifl Weaker dianfiQl- Weather LJ>ndei^n>iind- AccuWealhar 



Send teadbach 



NTRB. This is a "screenshot" or image of an 
interactive weather result. The buttons do not 
function in the screenshot. When rating, 
assume that users are able to see the 
temperature, precipitation, and wind for each 
day. 



Proprietary and Confidential - Copyright 2014 



85 



2. Do not mark a TLRB and a NTRB as duplicates of each other. As discussed, TLRB ratings are based 

primarily on the MC of the LP, whereas NTRB ratings are based primarily on the content and prominent links 
inside the result block. 



These Two Results Should Not be Considered Duplicates 


Explanation 


Leia Oraana Solo - Wookieepedia. the Star Wars Wiki 
starwa rs. wikia.com/wiki/Lei a_Org a na_So i o ' 

Her adoptive parents could not have children of their own, so Leia became ttie crown 
princess, and, as such, was forrnalty referred to as Itie Princess Leia ... 

Saba Sebatyne - Bail Preslor Organa - Slave Leia costume - Breha Organa 


TLRB. This block has a title link at the top. 


The daughter of Jedi Knight Anakin Skywalker and Senator Padm^ 
Amrdala of Naboo, Leia was the younger twin sister of Luke Skywalker, 
and, shortiy after her birth, she tiecame the adopted daughter of Baii 
Organa and Queen Breha of Alderaan, making her a Princess of 
Alderaan. 


NTRB. This block does not have a title link at 
the top. 


Leia Oraana Solo - Wookieecedla, the Star Wars Wiki 
starwar? .wi kia. com/wiki/Leia_Organa_Sato 









3. There may be a delay between when the rating task is created and when you actually rate the block, causing 
some information in NTRBs to be a few hours or even days out of date. Even though some NTRBs or SIRBs, 
such as stock price informational blocks or weather informational blocks, are designed to give users extremely 
recent information, don't penalize a NTRB or SIRB for being out of date. Assume that the blocks show current 
information for users. 



Examples of Time-Sensitive NTRBs 


Explanation 










Visa Inc 










NYSE: V- Jan 17 5:38 PM ET 










232.18 +10.41 (4.69%) 

After Hours: 232.08 -0.10 (-0.04%) 








NTRB. This is a "screenshot" or image of an 
interactive stock price result. The date shown 




Open 


222.05 




is the date this rating task was created. There 


^^^^.^-^ " • ' ~" "^^^^^"^^"^^ 


High 


232 26 




may be a delay before you are asked to rate 


230 

10am 11 12 1 2 a 4pm 


Law 

Voluma 
Avg Vol 
Iblitt Cap 


222.04 
9,586,108 
3.031.000 
147.78B 




it. Please assume that users see current 
information. 


id 5d 1m fim 1y max 




















Google Financs ■ Yahoo Finance - MSN Money 




Olsdaknar 
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Examples of Time-Sensitive NTRBs 



Alice, TX 

Friday S:00 PM 
Clear 



66 



Preciprtalion: 10% 
Humidiity; 20% 
Winrii: 1 5 mpb 



flFM 



11 PM 2MA 

Sun Mon 



i>AM 8 AM 11AM 2 PM 
TLtft Wart Tnii Tri 



70M r 73* 50- 75* 50 ' 82' 50* ftj" 43" 60* 
ThB WasDharCtuinnfll- ViWalhiflf UndwigraiHid -Aci:uWaa;hflf 



Sand r«adbAiJi 



Explanation 



NTRB. This is a "screenshot" or image of an 
interactive weather result. The date shown is 
the date this rating task was created. There 
may be a delay before you are asked to rate 
it. Please assume that users see current 
information. 



14.1 Vital (V) 



The Vital (V) rating is used for TLRBs in these very special situations: 

• There is a dominant interpretation and clear user intent to navigate to a specific website or webpage, and the 
LP of the TLRB is exactly what the user is looking for. 

• The dominant interpretation of the query is an entity (such as a person, place, business, restaurant, product, 
company, organization, etc.), and the LP of the TLRB is the official webpage associated with that entity. 

In both cases, the query must have a dominant interpretation. If there is no dominant interpretation, it is not possible to 
assign a V rating. 

The V rating is based on the LP of the title link. NTRBs cannot receive the V rating. 

Most V pages are very helpful; however, this is not a requirement for a V rating. Some V pages are "official," but not 
very helpful. 

To give a V rating, the landing page and website must "work" - the LP must load, function, and be maintained. 
Finally, when in doubt, don't use the V rating. 



14.1.1 Go Queries with Vital (V) Pages 



For Go queries, the V page is the page requested by the user. If the query "asks" for the homepage of a website, only 
the homepage gets the V rating. If the query "asks" for a subpage, only that particular subpage gets the V rating. 



Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[nytimes], English (US) 

Go to the New York 
Times online newspaper. 


The New York Times - Breaking News. World News & Multimedia 
w* nytimes com;' 

Find breaking news, multimedia, reviews & opinion on Wasnington. business, sports, 
movies, travel, books, jobs, education, real estate, cars & more. 


The New York Times 
homepage and target of the 
query. 


1 1 . 1 ■ 1 ' 1 ' 1 ' 1 ■ 1 ' f 
OT SR R U V 
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Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[nytimes sports], English 
(US) 

Go to the sports section 
of the New Yorl< Times 
online newspaper. 


Sports News - The New York Times 

wmv. nyti m e s X Qm/pa g es/s po rts/i n d e.>:- htm 1 

Find breaking news and sports news on the NFL. the NBA. the NCAA, the NHL. baseball, 
golf, tennis, soccer, the World Series, Super Bowl, the Olympics and ... 


The sports section page of the 
New York Times and target of 
the query. 


OT SR R U V 


[walmart.com], English 
(US) 

Go to the official 
homepage of the 
Walmart online retail site. 


Walmart.com: Save monev. Live better. 
wmvwalmart.com/ 

Online shopping for the largest selection of electronics, home furnishings, video games, 
baby gear and more. Shop online and save money to live better, ... 


The Walmart homepage and 
target of the query. 


OT SR R U V 


[walmart store finder], 
English (US) 

Go to the store finder 
page on the Walmart 
website. 


Walmart.com - Store Locator 

iivwiv,walmart,com/storeLocator'ca_5torefinder.dci 

About Walmart,com Terms of Use Affiliate Program Sponsorship Submission 
International Customers About Our Ads Store Finder Printable Coupons Associate ... 


The Walmart store finder page 
and target of the query. 


OT SR R U V 


[lady gaga twitter], 
Enolish (US) 

Go to Lady Gaga's 
official Twitter page. 


Lady Gaga (ladygagal on Twitter 

https://lwitter.comyladygaga 

The latest from Lady Gaga (@ladygaga). 


Lady Gaga's Twitter page and 
target of the query. 


1 1 1 1 . 1 . 1 ' 1 . 1 ■ f 

OT SR R U V 


[google maps], English 
(US) 

Go to Google Maps. 


welcome 

maps. google. com..' 

FinrI local hiiQinpQ'ipc; vlpw matTn anri npt clrlvlnn rilrpctif>nc; in Gccal^ Pu'lar^s 

1 II lU luUul k/Ui^ll It7i!,i>t7i?. Vlt^fV llluLJi^ ul lU Ut^l. Ul IVII lU Ull t^UllU^I Ii3 II 1 ' UUUIE IVIuLVi^. 


The Google Maps site and 
target of the query. 

Note that even though the title 
link just says "welcome," the 
rating is Vital. 


1 1 1 1 1 1 1 1 f 

OT SR R U V 



14.1.2 Entity Queries with Vital Pages 



For entity queries, the official homepage of the entity is Vital. Some entities show a "splash page" to announce 
something to visitors to their website. A splash page should be rated V if the page behind it is V. 



Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[Madonna], English (US) 

Go to the homepage of 
Madonna's official 
website or find 
information about her. 


Madonna.com > Home 
mmv,madonna,:om/ 

Madonna's official web site and fan club, featuring news, photos, concert ticl<ets. 
merchandise, and more. 


Homepage of Madonna's 
official website. 


OT SR R J V 


[Gary Danko], English 
(US) 

Go to the homepage of 
the Gary Danko 
restaurant's website or 
find information about the 
restaurant. 


Welcome to Restaurant Gary Danko 

mm garydanfco.com.1'' 

Gary Danko, tine award-winning chef who has come to epitomize sophisticated 
contemporary cuisine, has opened Gary Danko. 


Homepage of Gary Danko 
restaurant. 


OT SR R U V 
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Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[Bourne Legacy], English 
(US) 

Go to the movie's official 
homepage or find 
information about it. 


The Bourne Legacy - Official Site ] Own It NOW on Blu-rav, DVD a... 
■,v,V'V :hebournelegacy com/ 

The Bourne Legacy starring Jeremy Renner. Rachel Welsz. Edward Nortoh, Stacy 
Keach, arid Oscar Isaac cohtlhues the thrilllhg e.»:perience of Jasoh Bourhe. 


Homepage of the movie (on 
the movie studio's website). 


OT SR R J V 


[Maytag], English (US) 

Go to the homepage of 
this appliance 
manufacturer or find 
information about the 
brand. 


Maytag | Kitchen, Laundry & Home Appliances 
vftw.' maytag com; 

Shop Maytag washers, clothes dryers, refrigerators, ranges, dishwashers, water filters 
arid accessor es direct from Maytag 


Homepage of the IVIaytag 
website. 


OT SR R U V 


[ipod nano], English (US) 

Go to the iPod nano 
page on the Apple 
website or find 
information about it. 


Apple - iPod nano with Multi-Touch. 

wm. a ppl e. c om.i'ipo d -n a n O.I 

iPod nano features a Multi-Touch display, bullt-ln Bluetooth. FM radio, video, and more. 
Just $149. 


IPod nano page on the Apple 
website. 


OT SR R U V 


[Baseball hall of fame], 
English (US) 

Go to the Baseball Hall 
of Fame website or find 
information about it 


Baseball Hall of Fame | Preserving Historic - Honoring Excellence ... 
baseballhall.org/ 

The official site, of the National Baseball Hall of Fame In Cooperstown. New York. 


Homepage of the Baseball Hall 
of Fame website. 


1 " t II t'-'Hi- t"'l""'f 

OT BR R U V 


[IVIasters Golf 
Tournament], English 
(US) 

Go to the Masters 
website or find 
information about it. 


Welcome to the 2012 Masters Golf Tournament 

Wivw.masters.com/ 

The Off Dial Site of the Masters Golf Tournament 201 2. This major golf tournament is 
played annually at the Augusta National Golf Club. 


Homepage of the IVIasters Golf 
Tournament website. 


OT SR R U V 


[Freal<onomics blog], 
English (US) 

Go to the Freal<onomics 
Blog. 


Freakonomlcs » Blog 

mvx freakonomlcs. com/blog./ 

This paper Investigates whether demand-side rrarket pressure explains colleges' 
decisions to provide consumption amenities to their students. We estimate a ... 


Freal<onomics Blog page on 
the Freakonomlcs website. 


OT SR R u v 


[Harvard], English (US) 

Go to the Harvard 
University website or find 
information about it. 


Harvard Universitv 

WMV.harvard.edu/ 

Harvard University, which celebrated its 375th anniversary In 2011, is the oldest 
institution of higher learning in the United States. Harvard University is devoted ... 


Hnmpnanp nf thp Harvard 
University website. 


1 i < . ,t In,!, 1 1 f 

OT SR R U V 



14.1 .3 Queries with No Vital Result 

Most queries do not have a Vital result. Here are situations for which there is no V result. 

• The query does not have a dominant interpretation. 

• The query is not an entity or is not a navigation query. 

• No official website or webpage exists for the entity. 

• No person or entity can "own" the topic of the query. 
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Query 


Vital Result? 


Description 


[ADA], 
English (US) 


No V result is 
possible. 


There is no dominant interpretation. The following entities are all common 
interpretations. Each interpretation has an official homepage, but none is V since there 
is no dominant interpretation. 

• Americans with Disabilities Act 

• American Dental Association 

• American Diabetes Association 


[knitting], 

1 — 1 : _ i_ /I 1 o \ 

English (US) 


No V result is 
possible. 


This is an information query. Knitting is an activity anyone can do and that anyone can 
create a website for. There is no one official source for knitting information. No one 
can own this topic. 


[diabetes], English 


No V result is 

nnQQihlp 

jJUoolUIC 


This is an information query. No person or entity can claim ownership of the query 

rHiahptPQl 

[UIClUdC?Oj ■ 


[ipod reviews], 
Enalish fUS^ 


No V result is 


The query [ipod] English (US) has a Vital result, but [ipod reviews] does not. [ipod 
reviews] is a Know query. Users are looking for information that many sites can 
provide. 


[how old is britney 
spears?], English (US) 


No V result is 
possible. 


The query [Britney Spears] is an entity query, but [how old is britney spears] is not. 
This is a Know query. Users are looking for information that many sites can provide. 



Be careful with queries such as [diabetes], [knitting], [furniture], [cars], [toys], etc. There is no Vital result possible 
because no one can "own" the topic of these queries. TLRBs with URLs that match the query may not even be helpful. 

Sites claiming to be official may not actually be official sites. The V rating should NOT be assigned on the basis of the 
result block or appearance of the URL alone. You must visit the LP and examine the website. Here are some 
examples of result blocks that may look Vital, but are not. 



Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[Ashley Tisdale], English 
(US) 


AshleyTisdale.org ~ Your first and best source for evervthing Ashle. .. 

ashleytisdale eg 

Vi'elconieto AshleyTlsdale org! ... AshleyTlsdale org & Finders Keepers Giveaway ... 
AshleyTisdale.org. Ashley Tisdale & Pianet Biue Doubie-Prize Giveaway ... 


The LP is not an official 
homepage for Ashley Tisdale; 
it is a fan site. Here is the 
"real" official V result: official 
Ashlev Tisdale website. When 
you are unsure if a celebrity 
site is official, please explore 
the site. You will usually be 
able to find information telling 
you that the site is a fan site or 
the official site. 


Go to this celebrity's 
official website or find 
information about her. 


or SR R U V 


[Branson, Missouri], 
English (US) 


Branson com : The Official Branson Website 
w^™.branson.com/ 

Branson IVlO's Offciai Resource Center for Visitors, Locais. Discounts. Shows, Lodging. 
Restaurants. Vacations. Maps, News, and Events. 


The title of the TLRB and the 
LP display the words 
"Branson.com Official 
Website," but it is the 
homepage of the Branson.com 
website, not the homepage of 
the official city of Branson, 
Missouri website. Here is the 
official Vital result for the citv of 
Branson, Missouri. 


Find information about 
this city or go to its 
official page. 


OT SR R U V 
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Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[fabric], English (US) 
Purchase fabric online. 


Fabric - Discount Fabric - Apparel Fabric - Home Decor Fabric ... 

mw.fataric.com/ 

Fabric online from discount to designer. Fabric oniine Fabric Store - Here' you'ii Find 
Quiitiiig Fabric iike mcxla fabric. Fashion Fabric iike minKy fabric. iHome ... 


No V result is possible for this 
query because no one can 
claim ownership of it. Many 
websites sell fabric online. 
The LP is the homepage of a 
well-known, reputable fabric 
website and would be helpful 
for many or most users. 


or 5R R LT V 


[chairs], English (US) 

Purchase chairs online or 
find information about 
chairs. 


Chairs.com 

\\m\i chairs com.f 

This is an examole of a fake search oaae (see Section 7.3.2) and is 
considered Lowest quality. 


No V result is possible for this 
query because no one can 
claim ownership of it. Many 
websites sell and have 
information about chairs. This 
is a fake search page and 
Lowest quality, so should be 
rated OT. See Section 7.3.2 
for more information about 
fake search pages. 


¥ 1 1 1 1 1 ) 1 t 

OT 3R R Lf V 



14.1.4 Vital Pages for People Queries 

This section describes the use of the Vital rating for queries such as [oprah], [barack obama], and [lady gaga], which 
are names of people. This section does not apply to queries which include both a name and other words, such as 
[lady gaga twitter]. 

For a query that is the name of a real (non-fictional) living person, the V rating should be used when: 

• The query has a clear dominant interpretation, i.e., most people are looking for information about one 
particular person. For example [madonna], [barack obama], [shaquille o'neal]. 

• The result is the homepage of the person's official website, if such a website exists. 

Many or most people queries do not have a dominant interpretation. Queries like [ben smith], [mary jones], [elizabeth 
tucker], [susan greene], [paul richards], and [chad hancock] can have no V result because different users may be 
looking for different people. 

Even unusual sounding name queries may not have a dominant interpretation. For example, the queries [sam wen], 
[tran nguyen], and [david mease] can have no V result because there are multiple people with each of these names 
and it is not clear that most users are looking for any one particular individual. 

We will consider a person's website official if it is created by the person or an authorized agent of that person. The 
website must be maintained and have information or content which establishes that the website officially represents 
the person. This is a very high standard. When in doubt, do not use the V rating. 

Important: Websites that are under construction or obviously unmaintained should not be rated V, even if they were at 
one point created by or authorized by the person in the query. Please consider a personal website to be unmaintained 
if there is old or stale information displayed prominently. For personal websites which are generally very frequently 
updated, please look for updates within the last four months. If the personal website feels unmaintained, do not use 
the V rating. 
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Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 




TlgerWoods.com: The Official Site ofTiger Woods 

wwiv.ti g e rwo od s . c om/ 

Tiger's official site, in Engiish and Japanese, has news updates, goif tips, game 
schedules and an oniine ciub. 


Homepage of Tiger Woods' 
official website. 


[tiger woods], English 
(US) 


OT SR R U V 


Find information, news, 
etc. about Tiger Woods 
or go to tlie homepage of 
his official website. 






Tiger Woods (TigerWoods) on Twitter 

https -//twitter, c am 'Ti g e rWo 0 d s 

tiger Woods. @TigerWoods. Off cial Twitter account of Tiger Woods. Jupiter Island. 
FL ■ http*m«i. TigerWoods.com ■ 277 Tweets ■ 16 Following ■ 2,924,010 ... 


Thi^ 1*% Tinpr Wnnd^' Twittpr 
page. It Is not his official 
website. 






This result Is not Vital. 




<3T SR R U V 






Oprah WInfrev's Official Website - Live Your Best Life - Oprah.com 

Oprali,com is the official wettsite for everything in Oprali's world. Find advice on your 
health, t>eaijty, cooking and recipes, money, decorating, relationships and ... 


Homepage of Oprah's official 
website. 


[oprah], English (US) 


riT R 11 \t 

\j 1 n u V 




Find information, news, 
etc. about Oprah Winfrey 
or go to the homepage of 
her official website. 






Oprah-Wlnfrey.com 

wmv- oprah -Winfrey. com.i'' 

Eva Cassidy: True Talent Cannot Be Silenced ■ Remaining Doors Pay Homage To Jim 
Morrison On 60th Birthday ■ Photographs And Links To 20th Century ... 


1 [Mb lb lIUL V^pidll b UIIILfldl 

website, even though the URL 






(oprah-winfrey.com) matches 
her name. 




OT 3R R U V 






Emma Watson The Official Website 

WiV/\' emniav/atson com/ 

Emma Watson's off cial website. ... At 22 years old Emma is already an accomplished 
actress having appeared in 1 2 films. Click here to read her full biography. 


Homepage of Emma Watson's 


[emma watson], English 
(US) 


OT SR R J V 


official website. 


Find information, news, 
etc. about Emma Watson 
or go to the homepage of 
her official website. 






Emma Watson | Facebook 

www.f a cebookcom/emmaw/atson 

Emma Watson. London. United Kingdom. 7205161 likes ■ 388915 talking about this. 


XhiQ iQ a \A/fill-mpintainpH 
1 iiio lo a vvwii iiidiii idi 1 idj 

Facebool< page but it is not 
Emma Watson's official 






website. 




OT SR R Li V 


This result Is not Vital. 
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Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[sam wen], English (US) 

Find information, news, 
etc. about someone 
named Sam Wen or go 
to the homepage of his 
official website. 


Sam Wen 

WAW.samwen com/ 

My name is Sam Wen. and 1 work at Square in San Francisco, in the past, i was a 
defense industry junltie. Atter i recovered, i joined Jack Dorsey. Jim McKeivey. ... 


There are multiple people with 
this name. It is not clear which 
Sam Wen users may be 
looking for. There is no 
dominant interpretation, and 
therefore no V result is 
possible for this query. 


OT SR R U V 


[amanda bynes], English 
(US) 

Find information, news, 
etc. about Amanda 

D\/noc r\r r\r\ \r\ tho 
Dyllco Ul yU lU Lllc 

homepage of her official 
website. 


Amanda Bvnes 

w^^. amandabynes.com/ 

Under Construction? Copyright © 2000. All Rights Reserved. 
This result is not Vital. 


Amanda Bynes is an 
actress. The LP displays an 
"Under Construction" 
message. The copyright date 
shown is 2000. There is no 

rT\ntont r\Ti tho r\ q n o that 
OUiUuiU Ull Lllu pdyu LilaL 

establishes this as her official 
website. 


or SR H u V 



14.1.5 Vital Pages for URL Queries 

Some queries look like URLs. We will call these queries URL Queries. Here are some types of URL queries: 

• Exact, perfectly-formed, working URLs, such as [http://www.ibm.com] or [www.ibm.com] or [ibm.com]. 

• Imperfect URL queries: Queries that look like URL queries, but are not "working URL" queries. These URLs 
do not load if you type or paste them into your browser address bar. Even so, we believe users have a 
specific page in mind. 

Some queries, such as [yahoo] or [yahoo mail] are website names. These queries do not contain ".com" or "www" or 
other standard components of a URL. These are Go queries, but we will not consider them URL queries. 



Assign a V rating to results for URL queries based on user intent. Assign a V rating only if there is little or no doubt 
that the page matches user intent. 



Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[www.myspace.com], 
English (US) 


Myspace { Social Entertainment 

w»- mys pa ce - c om..' 

Myspace Is the leading social entertainment destination powered by tne passion of fans. 
Music, movies, celebs, TV. and games made social. 


Homepage of the IVIySpace 
website. 

This is a perfectly-formed 
working URL query. The user 
intent is clear: to go to the 
MySpace website. 


OT SR R J V 


[www. yahoo. cOm], 
English (US) 

[yahoo.xcom], English 
(US) 


Yahoo! 

wwA'.ya hoo.com/ 

Welcome to Yahoo!, the world's most visited home page. Quickly And what you're 
searching for. get in touch with friends and stay in-the-know with the latest ... 


Homepage of the Yahoo 
website. 

Even though these queries are 
imperfect URL queries, it's 
clear the user wants to go to 
the Yahoo website. 


OT SR R U V 
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Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[huffintonpost.com], 
English (US) 

This query is a worl<ing 
URL, but it is clearly 
mistyped. The clear user 
intent is to navigate to 
huffingtonpost.com. 


huffintonpost.com 

w.y: oost-com/ 

huffintonpost com ... Shopping|Blacht Voices|Politics News|Japan|FaceboDk. Copyright 
O 2012 hufflntonpostcom. All Rights Reserved. Home | Legal. 


Homepage of the 
huffintonpost.com website. 

This result is nof Vital. It is not 
the result the user is looking 
for. This is a fake search page 
and Lowest quality, so should 
be rated OT. See Section 
7.3.2 for more information 
about fake search pages. 


OT SR R U V 


Breaking News and Opinion on The Huffington Post 

"'■>■"■ h uffi n gto n po St . c om/ 

The destihatioh for hews, blogs ahd origihal cohteht offerihg coverage of US politics, 
ehtertaihmeht. style, world hews, techhology ahd comedy. 


Homepage of the Huffington 
Post website. 

The clear user intent is to 
navigate to 

huffingtonpost.com. This 
result is Vital. 


OT SR R J V 


[wwww.ibm.com], 
English (US) 

Even though the URL 
doesn't load, it is clear 
that the user wants to go 
to the IBM homepage. 


IBM - United States 

mw.ibm.com./ 

The IBM corporate home page, ehtry poiht to informatioh about IBM products ahd 
services. 


Homepage of the IBM website. 


OT SR R J V 


[tax form 1 040 irs.gov], 
English (US) 

Even though the query 
contains spaces, it is 
clear the user wants to 
go to the page on the 
official IRS government 
website for the current 
1040 tax form. 


[PDF] Form 1040 - Internal Revenue Service 

viratw. i rs . g 0 v/pu b/irs -pdfi'fl 040 . pdf 

File Format: PDF/Adobe Acrobat - Quick View 

Form 1040 Departmeht of the Treasury — Ihterhal Revehue Service. 0MB No. 1545- 
0074. (99). IRS Use Ohiy — Do hot write or staple ih this space. U.S. Ihdividual ... 


Page with the 1 040 tax form 
on the official IRS website. 


or SR R u v 


[i hire chemists.com], 
English (US) 

Even though the query 
contains spaces, it is 
clear the user wants to 
go to the iHireChemists 
website. 


Chemistry Jobs. Careers | IHireChemists 

www.ihirechemists.com/ 

Fihd 3683 chemistry jobs listed ih your area oh IHireChemists Browse curreht listihgs. 
register ahd apply ohiihe today. 


Homepage of the 
IHireChemists website. 


1 1 1 1 1 1 ' t ' 1 ■ ¥ 

OT SR R U V 



14.1.6 Vital Pages and Geographic Location 

Some international entities Inave multiple versions of tineir Inomepages for different languages or countries. A 
Inomepage sinould be rated Vital winen: 

• TInere is clear user intent to navigate to a specific website or webpage, and the landing page is exactly what 
the user is looking for. 

• There are multiple versions of the official page, and the page seems right for users in the locale. 
Otherwise, please assign a rating based on utility for the user. 
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Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[United Nations], Frencli 
(FR) 

Go to ttie United Nations 
website. Frencti users 
expect to go to tlie 
version of tlie website 
tliat is appropriate for 
Frencli speal<ing users. 


Bienvenue aux Nations Unies 

un.org/fr/ 

Nous, peuples des Nations Unies ... une ONU plus forte pour un morde meilleur. Paix 
et securite - Developpement - Droits de I'homme - Action humanitaire ... 
Travailler a I'Organisation des ... - A propos de I'ONU - Objectifs du P^^lillenaire pour 


Homepage of fine Frencti 
version of tlie United Nations 
website. 

Ttie Frencti version of ttie 
website is V for all Frencti tasl< 
tocations. It is ttie rigtit official 
page for Frencti speal<ing 
users. 


OT 3R R U V 


[United Nations], Russian 
(RU) 

Go to ttie United Nations 
website. Russian users 
expect to go to tlie 
appropriate version of 
the website. 


Welcome to tlie United Nations: It's Your World 

w/fw.un.org/ - Translate this page 

sjiiji ^S,n Ij-g-g United Nations Nations Unies OpraHH3aL(Mq OBteflMHeHHbij; Ham-iii 
Las Naciones Unidas. 


Ttie United Nations website tias 
six versions of its website: 
Arabic, Japanese, Englisti, 
Frencti, Russian, and 
Spanisti. Ttie Russian version 
of ttie website would be V for 
ttiis query. Ttiis "ctioose your 
language" page is not. 


or BR R U V 


[un.org], 
Russian (RU) 

Go to un.org. 


Welcome to tlie United Nations: It's Your World 
www.un.org/ - Translate this page 

sjaaji Ij-g-g United Nations Nations Unies OpraHM3aL(Mq OBteflMHeHMbu; Ham-iii 
Las Naciones Unidas. 


"Choose your language" page 
on the United Nations website. 

^inr'O tho 1 P ic tho r»3no "acLfoH 
on lou u It; 1— 1 lo Lilt; jjciyt; a.or\t;u 

for" in the query, it is V, even 
though a Russian version of the 
United Nations website exists. 


OT SR R LI V 



14.2 Useful (U) 



A rating of Useful is assigned to title link result blocks (TLRBs) and non-title link result blocks (NTRBs) that are very 
helpful for many or most users. U results are highly satisfying and a good "fit" for the query. In addition, they often 
have some or all of the following characteristics: high quality, authoritative, entertaining, and/or recent (such as 
breaking news on a topic). 

A query can have many Useful results. 

Have high standards for using the U rating. This is especially important for queries with many, many on-topic results. 
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14.2.1 Examples of Useful (U) Result Blocks 



Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[pleasant grove high 
school], English (US) 

Navigate to the 
homepage of a high 
school named "Pleasant 
Grove High School" or 
find information about it. 
This is a Go or Know 
query. 


Pleasant Grove Hiah School 

pghs,alpineschools,Qrg/' 

Pleasant Grove High School - Main Offce: (801) 610-8170 - Counseling: (801) 610- 
8171 - Attendance: (801) 610-8172 700 East 200 South Pieasant Grove, UT ... 


This query does not have a 
dominant interpretation. 

Each of these LPs is the 
homepage of a high school in 
the U.S. with this name. The 
high schools are located in the 
states of Utah, California, 
Alabama, and Texas. 

Since there appears to be only 
four high schools in the country 
with this name, the homepage 
of each school is a good match 
for the query and merits a U 
rating. 


OT SR R U V 


Pleasant Grovs High School: Horme Page 
pghs.schoolloop.com- 

With the heip of Capitoi Region high school students, we wiii recognize teachers, 
administrators, schooi programs, and business partners for e.i((;eiience. 


OT BR R Lf V 


pleasantgrovehigh.jefcoed com; 

Sh;ip Navigation Linte Pieasant Grove HE .... Pleasant Grove High School. The mission 
of our school Is to develop capable students who will become ... 


OT SR R U V 


Pleasant Grove High SchooJ / Overview 

www.pgiscl.net''Domain/32 " 

Welcome to Pleasant Grove High School. High school is a time of change for ... 


OT SR R LF V 


[poison ivy], English (US) 

Find information about 
poison ivy. This is a 
Know query. 


Outsmarting Poison Ivy and Other Poisonous Plants 

invw.fda g: . : ; : ' ; .imersjxonsumerupdates,iucm049342.htm 

Jump to No; Contaqious: Poison ivy and other poison plant rashes can't be spread 

from person to person. But it is possible to pick up the rash from plant ... 


Page on an authoritative 
website that answers this 
question very well. This result 
would be very helpful for many 
or most users. 

There are many, many on-topic 
results for this query. This 
result merits a U rating. 


OT R J V 


[IVIanresa reviews], 
English (US) 

Read reviews for this 
restaurant. This is a 
Know query. 


Manresa, Los Gatos - Restaurant Reviews - TrIpAdvlsor 

wmir.tripadvisor.com > ... > Los Gatos > Los Gatos Restaurants 
•kifk-ki Rating: 4.5 - 79 reviews 

Manresa. Los Gatos: See 79 unbiased reviews of Manresa rated 4.5 of 5 on 
TrIpAdvlsor and ranted #4 of 97 restaurants in Los Gatos 


rage on a weii-Known weusiTe 
with over 75 reviews for this 
restaurant. This page would be 
very helpful for many or most 
users. 


OT SR R Lf V 


[broadway tickets], 
tngiisn (Uo) 

Purchase tickets to a 
Broadway show. This is 
a Do query. 


Broadway tickets, show dates, calendar. Official TIcketmaster site. 

wvx ticketmaster.com/ta roadway 

Find and buy broadway show tickets at Ticketmaster.com. 
Broadway - Wicked (NY) Tickets - Evita (NY) Tickets - OtT-Broadway 


Page on a reputable ticket site 
to complete this transaction. 
1 nis page wouia oe very neiprui 
for many or most users. 

There are many, many on-topIc 
results for this query. This 
result merits a U rating. 


OT SR R U V 
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Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[csco], 
English (US) 

FinH citofk niintp 

1 II IVJ OLVJLriX V_JUVJLC 

information for Cisco. 
Tfiis is a Know query. 


oummarv lor Lfisco ovstcFTIS. inc.- TanuO! rinancc 

finance. yahoD-Com/q?s=C SCO 

View the basic CSCO stock chart on Yahoo! Finance. Change the date range, chart type 
and compare Cisco Systems. Inc against other companies. 
JNPR - News - CSCO Interactive Chart - Key Statistics 


CSCO is the stock symbol for 
the Cisco Corporation. This 
page would be very helpful for 
many or most users. 

There are many, many on-topic 
results for this query. This 
result merits a U rating. 


OT SR R 


J V 


[I<ids bacl<pacl<s], Englisfi 
(US) 

Find information about 
l<ids bacl<pacl<s or 
purchase a l<ids 
bacl<pacl<. This is a Do 
or Know query. 


Kids' School Backpacks & Messenger Bags | Free Shipping from L... 

iww.llbean. com.i'ilb.i'sh op/81 E?page=backpacks 

Free Shipping. From pre-Kto post-Grad. L.LBean school backpacks and messenger 
bags earn high marks from students & parents allte. Our durabie kids' ... 


LL Bean is well known for 
manufacturing and selling high 
quality backpacks for kids of 
different ages. There is a lot of 
product information and many 
user reviews for each backpack. 
This page would be very helpful 
for many or most users. 

There are many, many on-topic 
results for this query. This 
result merits a U rating. 


OT SR R 


U V 


[ikea], English (US) 

Go to the official Ikea 
homepage for U.S. 
users. This is a Go 
query. 


Weicome to IKEA.com - IKEA 

wvm.ikea.comj' 

Featuring Scandinavian modem styie furniture and accessones Include storage options, 
lighting, decor products, hltchen appliances and pet care. 


The MC allows users to select 
their location. Users in the U.S. 
would click the "United States" 
link to get to the appropriate 
homepage. 

Because this store exists in 
many different countries, a 
"choose your location" page is 
very helpful for most users. 


OT SR R 


U V 


[every breath you take 
lyrics], English (US) 

Find the lyrics to this 
song written by Sting. 
This is a Know query. 


Sting.com - Lvrics : Every Breath You Take 

'imx- sti n g .c om.i'd 1 sc og ra phy/lyrl c s.'lyri c.i'son g/1 3 0 

Every breath you take And every move you make / Every bond you break, every step 
you take i I'll be watching you Every single day / And every word you say ... 


Page on the official Sting 
website with the requested 
lyrics. There are many low 
quality lyrics results, but we can 
have confidence in the accuracy 
of the lyrics on this site. 

There are many, many on-topic 
results for this query. This 
result merits a U rating. 


OT SR R 


U V 


[dance video], English 
(US) 

Find a dance video to 
watch. There are many 
good, entertaining, and 
popular dance videos on 
video websites. Users 
are looking for good or 
entertaining dance 
videos. This is a Do 
query. 




Evoiution of Dance - Bv Judson Laippiv - YouTube 


This is a very popular video of a 
comedian demonstrating dance 
styles from previous decades. 

There are many, many on-topic 
results for this query. This 
result merits a U rating. 




'jWAryoutube.comi'watch?v=dMHQbHeiRNg 
Apr 6. 2006 - Uploaded byjudsonialpply 

Evoluticn of Dance - By Judson Lalpply. judsonlalpply 5 videos 
Subscnbe SutDscnbed Unsubscribe 63.628 ... 


OT SR R 


LP V 
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Query and User Intent 



Result Block and LP, Utility Rating 



Description of the LP 



Images of purple coneflower 

bing.coFn'images 



[purple coneflowers], 
English (US) 

Find images of or 
information about purple 
coneflowers. 




See mere than 47.400 images 



OT SR 



This is a TLRB, even though it 
has special content. 

There are helpful images in the 
result block, as well as many, 
many additional helpful images 
on the LP. 



[1600 amphitheatre 
parkway. Mountain View, 
CA], English (US) 

Users may be looking for 
directions, a map, 
information about what 
business resides at this 
address, etc. 













' r 












1 


\ 

i 


I IT" 

u 






s 

i I 


1 J 




l&Dl^ An-iptiitheatra Pkwy Mountain View, CA 94043 


1 



OT SR 



This is a NTRB, so the rating 
should be based primarily on 
the content in the block. In 
addition to the map, the result 
block also contains helpful links 
to get directions and explore 
public transportation options. 




[john boehner], English 
(US) 

Users may be looking for 
news, biographical 
information, images, etc. 



John Boehner 

iiTj'.' ■ J !■ - ■..( ';■■■.: i-ljlii l-inrtB nF RspstFtililr,"! 1 

St<*s Hfw of Rtpiwilnlim A mflrpbar of Ihj- Rt^ubkan 
islhtUS R4|m£a<Mr*lhxilOhki't Rh caAgreEMnalilHtri 
■iWtcv WibtAdia 

Dam: tiyMrrbAi 17. 1919 U). Rdadn^ OH 

SfKMH*: Debta Bothnii l>n 19J3|' 

OFfko; R«pw>}m?li>^ \R'W Vlh Pistrid) ^nco SUM 

Pal^nb; Eat Hmv SntHiar. P^arv Amv I 

PivWdoi ofllvn; fAngiHv i»dn |24D7-2eit>, UliMy tatdn 

(Tme-iTHlJI 



GiA updalL's abod 




Rgcenl pasta 






1 Ihtfin-HdcirTl has (fl 




1 ta\tiijsn^'\aiai^*t 



PenplB a\so ssarchr for 



Si. SI 12 



1 1 any Kr^i Hnch 



This is a NTRB, so the rating 
should be based primarily on 
the content in the block. 

There is a lot of content at a 
glance to make it easy to easily 
understand who John Boehner 
is, including biographical 
information, images, recent 
posts, etc. 

There are also several helpful 
links to authoritative sources, 
which add to the overall 
experience of the result block. 
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14.3 Relevant (R) 

A rating of Relevant is assigned to results tinat are Inelpful for many users or very Inelpful for some users. 

R results have fewer valuable attributes than U results. R results should still "fit" the query, but they might be less 
comprehensive, less up-to-date, come from a less authoritative source, etc. 

R results generally are not egregiously low quality, out-of-date, or inaccurate. R results are average to good. 

The R rating can apply to non-title link result blocks (NTRBs) as well. A R NTRB may not be highly satisfying in some 
way, or it may serve a minor interpretation. 



14.3.1 Examples of Relevant (R) Pages 



Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[map of Seoul, korea], 
English (US) 

Travel to Seoul or find 
current information about 
the city. 


Map of Seoul 

w/f. lonelyplanet.com/maps/asia/south-korea/seoul/ 

Map of Seoul and travel information about Seoul brought to you by Loneiy Pianet. 


rage wiin a map ot tne ciiy ot 
Seoul. This page would be 
helpful for some users. 

There are many, many maps of 
Seoul with more detailed 
information covering a wider 
area of the city. This page is 
not helpful enough for a rating 
of U. 


OT SR R U V 


[Tom Cruise], English 
(US) 

Find biographical 
information or current 
news/celebrity gossip 
about Tom Cruise. 


Tom Cruise Pictures. Biographv. Filmographv. News ... - Starpulse... 

mw.starpulse-com > Actors > C 

Tom Cruise Pictures, Biography. Filmography. News. Great Fiim Moments. Videos. 


A page of information about 
Tom Cruise. This page would 
be helpful for many users or 
very helpful for some users. 

There are many, many pages 
about Tom Cruise. This page is 
not helpful enough for a rating 
of U. 


OT SR R U V 


[abe lincoln's birthday], 
English (US) 

Find this specific piece of 
information. 


List of Presidents of the United States bv date of birth - Wikipedia .„ 

e - i g- ■_!sl_of_Presiclents_of_the_Uniteci_.?^ _ 

1 Birthday. 2 Months: 3 Generation: 4 Born on the same date 5 Most and Abraham 

Lincoln and his first Vice President Hannibal Hamiin were Doth born in ... 


Wikipedia page that displays 
the birthdays of all U.S. 
presidents, including Abraham 
Lincoln. However, Lincoln's 
birthday is not prominently 
displayed. 


OT SR R U V 


[every breath you take 
lyrics], English (US) 

Find the lyrics to the 
song "Every Breath You 
Take," which was written 
by Sting. This is a Know 
query. 


STING LYRICS - Every Breath You Take 

mw aziyrics com/iyricsi'sting/everybreathyoutake htmi 

Every breath you take. Every move you make. Every bond you break. Every step you 
take i'il be watching you. Every singie day Every word you say ... 


Page on a lyrics website with 
the requested lyrics. There are 
many lyrics websites on the 
web. Often, pages with lyrics 
are not 1 00% accurate. R is an 
appropriate rating for average 
pages with the requested lyrics. 


OT SR R U V 
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Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[latin dance video], 
Englisti (US) 

Find a Latin dance video 
to watcti. Tliere are 
many good, entertaining, 
and popular Latin dance 
videos on video 
websites. Users are 
looking for good or 
entertaining Latin dance 
videos. 


Amazing Latin Dance! - YouTube 
^HS^^HW^ vvnv.yo utu be. com/watch ?v= FSzC GgZM a WQ 
^CP^W^^^ ^"9 ^^''^ - Uploaded by VisioneerMedia 
^^H^^j^^f You need Adobe Flash Player to watch this video. ... Sign In with 
^^fV^BjHM your YouTube Account (YouTube. Google-f ... 


There are many, many Latin 
dance videos. This video is 
entertaining and the dancers 
are good, but there are many 
non-professional videos of 
talented amateur dancers. 


OT SR R U V 


[Nicaraguan money], 
English (US) 

Find out what 
Nicaraguan currency is 


Nicaraguan cordoba 

ThecordDba is the currency of Nicaragua. II is divided into 1 DO centavos 
Wiidpectia 






This is a NTRB. While it gives 
some basic information and is 
helpful, it is not highly satisfying. 
For example, it doesn't show 








called, what it looks like, 
the exchange rate, etc. 


OT SR R U V 


what the currency looks like. 


[mercury], English (US) 
Find infnrmatinn ahniit 

1 III \A 1 1 1 1 w 1 1 1 1 d LI w 1 1 ciky w m l 

the Mercury car brand, 
the planet, or the 
chemical element. 


See results about 

Mercury 
Element 

Mercury Is a chemical element with the symbol Hg and 
atomic nijmE)er 80. It Is commonly known as ... 

^^"""^B Mercury 

f S Planet 

, ■ Mercury Is the Innermost planet In the Solar System. It Is 
^^^^^^^^^1 also the smallest, and Its orbit Is the most eccentric ... 




This is a NTRB. It helps users 
understand some different 

lllUdl III lyb Ul Lllo WUlU 

"Mprnirv " and thp link^; Iparl tn 

IVI^IWUIV; C4.IIU 11 1^ III ll\0 l\^U\_l L\J 

these SERPs: Element LP and 
Planet LP. However, it is 
missing at least one common 
interpretation: the car brand. 














OT SR R U V 






Shutterfly | CrunchBase Profile 

www.crunchbase.com > Companies 

As long as there have been people, there have been stones. People have always wanted 
to pass on their memories. 


A page on crunchbase.com with 
a profile of the Shutterfly 
company. 

The MC of the LP has a lot of 
helpful information about the 
company. 

This is a high quality page on a 
member site on the TechCrunch 
network. The LP has some 
helpful information about 
Shutterfly. Even though the LP 

ilclb a LU ol lULltJi 1 ly .OUl M III a 

logical spot on the page, the link 
does not play a large role in 
rating. This result merits a rating 
of R. 


[shutterfly], English (US) 

Go to the homepage of 
the Shutterfly website. 


OT SR R U V 
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14.4 Slightly Relevant (SR) 



A rating of Slightly Relevant is assigned to results tinat are Inelpful for some or few users. SR results may serve a 
minor interpretation, be low quality, have stale (outdated) information, be too specific, too broad, etc. to receive a 
higher rating. 

The SR rating can apply to non-title link result blocks (NTRBs) as well. A NTRB may be too specific, have too little 
content, or serve a minor interpretation. 



14.4.1 Examples of Slightly Relevant (SR) Pages 



Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[lack of sex and 
problems with my 
marriage], English (US) 

Find help for marital 
issues. 


5 Tips to Fix a Sexless Marriage Or Relationship 

ezinearticles.com > Reiationships > Marriage 

Feb 25, 2DD8 - if you're in a sexless marriage or reiationship and wish to change it to 
the better, this article is def niteiy for you, and i urge you to read it through. 


This is a low quality article. The 
quality of writing is poor. The 
article is written by a person 
without expertise in marriage or 
relationship counseling. Even 
though the article is about the 
query, the page is low quality 
and untrustworthy. It would be 
helpful for few users. 


OT SR R U V 


[hot dog], English (US) 

Find information about 
hot dogs, such as 
recipes or nutrition 
information. 


Hot Dog... The Movie (19841 - IMDb 

mw, imdb,com;'titie;ltQ087425/ 

Directed by Peter Marhie, With David Naughton, Patrick iHouser, Tracy Smith. John 
Patrick Reger, Naive corn-pone sKiing wunderKind picKs up beautifui ... 


The movie "Hot Dog," which 
came out in 1984, is a minor 
interpretation. This IMDb page 
would be helpful for few users. 


OT 3R R U V 


[best golfers], English 
(US) 

Professional golfers are 
ranked based on money 
earned in tournaments. 
We assume users are 
interested in the most 
current Money List. 


Golf Stats Leaders - PGA Tour - 2009 - CBSSports.com 

'i™v,cbssports,CDm.'golf.'Stats./statsJeaders..'PGA.'20(I9.'careerMoneyL,,- 
Golf statistics for the PGA Tour, LPGA Tour, Champions Tour and more at 
CBSSports,com, ... Who is this year's money ieader? ... Eagies (Hoies Per), Greens iiit in 
Regulation. Longest Drives. Money List Par 3 ... Career Money Leaders - 2009 ... 


The landing page shows the 
2009 PGA Money List. It is 
outdated and stale for the query 
today. It would be helpful for 
few users. 


OT Sfi R U V 


[britney spears], English 
(US) 

Find current news or 
pictures related to 
Britney Spears. 


reviewjournal com - News: Britney Spears' 55-hour marriage annu. . 

mnv, revi ew]ou rn a i , c om/ivr]_h om e,j'2 0 0 4/„ „i'2 2 S 3 5 2 62 , htm 1 

Jan 6, 2004 - Men, don't give up on that fantasy just yet: Britney Spears is a single 
woman again. On Monday, a Claris County judge annulled Spears' ... 


2004 article about the 
annulment of Britney's first 
marriage. This is very old stale 
news that would be helpful for 
few users. 


or SR R U V 


[honda odyssey], English 
(US) 

We will assume that 
users are interested in 
the current Honda 
Odyssey model, unless 
specified otherwise. 


2010 Honda Odyssey - Kellev Blue Book 

vm,kt>b.conij'honda/odyssey;2010 honda-odyssey/ 

****t/ Rating: 8,B/10 - 531 reviews 

□ads will tell you the best vehicle to transport the wife and kids Is a big, burly SUV, but 
Moms know better. Despite its less-than-macho image, the minivan is stili ... 


The information on this Kelley 
Blue Book page about the 2010 
Honda Odyssey would be 
considered stale for the query 
today. It would be helpful for 
some or few users. 


OT SR R u v 
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Query and User Intent 



Result Block and LP, Utility Rating 



Description of the LP 



[abe lincoln's birthday], 
English (US) 

Find this specific piece of 
information. 



Why Abraham Lincoln's Birthday Isn'tA Federal Holiday- Busine... 

articles.businessinsicler.com > Coiiections 

Feb 13. 201 2 - This post originaiiy appeared at The Christian Science Monitor. Abraham 
Lincoln may be the greatest of ail US presidents. He ended siavery ... 



This landing page mentions the 
month and day, but not the 
year, of Lincoln's birth. Most 
users would be interested in 
also knowing the year. There is 
not enough information about 
the topic of the query. This 
page would be helpful for few 
users. 



[ebay], English (US) 

The dominant 
interpretation is to go to 
www.ebay.com. 



Ebay. com Site Info 

WAnA'.ale.ra.com > Shopping > Auctions 

ebay.com is one of the top TOO sites in the worid and is in the Auctions category. 



This landing page has 
information about web traffic to 
the ebay.com website. It would 
be helpful for few users. 



[every breath you take 
lyrics], English (US) 

Find the lyrics to the 
song "Every Breath You 
Take," which was written 
by Sting. This is an 
Know query. 



Sting - Every Breath You Take Lyrics 

wiTO.Iyricsmania com'every breathjou takejyrics stlng.htnil 

Every breath you take Every move you mate / Every bond you breaH Every step you 

tal^e / I'll be watching you .' Every single day Every word you say ... 



Even though this page does 
display the lyrics, it is a lower 
quality page than most lyrics 
pages. Notice the pop-up 
displayed on top of the lyrics 
and how the lyrics are split 
across two columns, as well as 
all the other distracting content 
on the page. It would be helpful 
for few users. 



See results about 



[the Vatican], English 
(US) 

Find information about 
the Vatican. 



Vatican City 

Vatican City, or Vatican City State, in Italian officiaily Stato 
della Citta del Vaticano, is a landlocked ... 



This is a NTRB. This result 
block does not have enough 
content to be satisfying. The LP 
for the Vatican City link in the 
result block is a SERP for 
[Vatican City]. This result is not 
very helpful overall. 



[another word for 
allowed], English (US) 

Find synonyms for the 
word "allowed." 



allowed past participle, past tense of al-low (Verb) 



Verb 

1 . Admit (an event or activity) as legal or acceptable. 

2. Give (someone) permission to do something. 



More info -Didionari'.com - Merriani-Webster-The Free Dictionary 



This is a NTRB. While this 
result block may help a few 
users think of another word for 
"allow," it is not very helpful 
overall. 



OT SR R U V 
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14.5 Off-Topic or Useless (OT) 

A rating of Off-Topic or Useless should be assigned to results that are helpful for no or very few users. OT results are 
unrelated to the query and/or have no utility. 

A rating of OT also applies when there is lack of attention to an aspect of the query that is important for satisfying user 
intent. 

The OT rating can also apply to NTRBs. An OT NTRB may be unrelated to the query, inaccurate, very unhelpful, or 
useless. 



14.5.1 Examples of Off-Topic or Useless (OT) Pages 



Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[Australian Open mens 
singles result 2008], 
English (US) 

Find a page that displays 
the 2008 men's singles 
result for this tennis 
tournament. 


2004 Australian Open - Men's Singles - Wlklpedla. the free ... 

en.wikipedia.org.'^A'il!i/'2004_Australian_Open_-_Men's_Singles 

Roger Federer defeated Marat Satin 7-6(7-3). 6-4. 6-2 in thefnai to win the Men's 

SIngies titie at the 2004 Australian Open tennis tournament. This victory ... 


Does not fit the user intent: 

This LP is about the 2004 
Australian Open, not the 2008 
Australian Open. It is OT 
because it does not pay 
attention to an aspect of the 
query that is important for 
satisfying user intent: "2008." It 
would be helpful for no or very 
few ij*?pr9 


OT SR R U V 


[german cars], English 
(US) 

Find information about 
German cars or go to the 
official homepage of a 
German automaker. 


Subaru Cars, Sedans, SUVs, Crossovers, Wagons | Subaru of Am... 

WW subaru.com/ 

Visrt Subaru of America for reviews, pricing and photos of Subaru Cars. Sedans. 
Wagons, Crossovers & SUVs. Find Subaru Deaier information. 


Does not fit the user intent: 

This LP is the homepage of 
Subaru, a Japanese car 
company, not a German car 


OT SR R Lf V 


company. It would be helpful 
for no or very few users. 


[anderson high school, 
austin] 

Go to the homepage of 
Anderson High School in 
Austin, Texas or get 
information about the 
school. 


Anderson Higii School 

w,v.foresthiiis.edUj''scliool_home.asp.»:?schooliD=1 

Congratuiations to Anderson High School senior vioiinist and Concertmaster Lindsey 
Suiiivan on being given the seat of Concertmaster of ... Lindsey is the frst Anderson 
student to receive concertmaster from OMEA Cincinnati. OiH 45255 ... 


Does not fit the user intent: 

This LP is the homepage of 
Anderson High School in 
Cincinnati, Ohio, not Anderson 
High School in Austin, Texas. 
It would be helpful for no users. 


T ■ 1 ' 1 ' 1 ' 1 - 1 ' 1 - 1 

OT SR R Lr V 


[opi blue nail polish], 
English (US) 

Find pages about the Opi 
brand's blue nail polish. 


Essie Blue and Green Nail Polish Shades - Zappos.com Free ... 

'imv.zappos.com > Beauty > Naiis > Essie 
Jan ID. 2012 

Blue and Green Nail Polish Shades by Essie at Zappos.com - 
Q FREE Shipping. ... Even then Csicos and ... 


Does not fit the user intent: 

This LP has the Essie brand of 
blue nail polish, not the Opi 
brand. It would be helpful for no 
or very few users. 


OT SR R Lf V 
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Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 


[company to get rid of the 
possum in my attic], 
English (US) 

nil lu cx ouiii)Jciiiy LU u ci|j 

and remove a possum 
from the attic. 


Complete Pest Control - Adelaide South Australia | Adelaide Pest ... 

c om pi etepest. c om .a u/ 

Get rid Df mice and rats from your home or business - caii 61 32 4000 today. Possum 
Removal. Not Just noisy - Possums can cause significant damage to you ... 


Does not fit the user intent: 

This LP is the homepage of a 
pest control company in 
Australia. U.S. users would 
need a U.S. company to take 

Ociio Ui Uiio [JIUUIUIII. 1 iioic; lo 

a mismatch between the page 
and the locale that makes the 
landing page OT. 


OT SR R U V 


[hot dog], English (US) 
Find information about 

IIUL UUyb, bUCII db 

recipes or nutrition 
information. 


Dog Houses: Design and Construction 

mw-peteducation,com.i'article-Cfm7c=2+209B8;aid=812 

Learn about the proper size design, buiiding materiais. construction, piacement. and 
bedding for dog houses. 


Keyword matches only: This 
LP has information about 

uuyiiuuouo cii lu 1 ici|j|Jci lo lu 

display the word hot. It is OT. 


OT SR R U V 


[tooth loss five years old], 
English (US) 

Find information about 

tnnth ln^<5 in p fivp-vpar- 
luuLi 1 ii.'oo II 1 CL live yccii 

old child. 


Gallery of Pennsylvania Fishes - Chapter 1 1 , Pikes 

mw-f is h - state .p a - u s..' p af i s h/f i sfi htm s.i'c h a p 1 1 pi l^es. htm 

Chapter 1 1 . PA Fishes book cover image - ciick for Tabie of Contents ... The northern 
pike is one of the few fish whose natural range Includes both North America and Eurasia. 
... in length, to the mighty muskeiiunge. whose state angling record is over 54 pounds. 
.... There Is no seasonal tooth loss that keeps them from biting. 


Keyword matches only: This 
LP has information about 
tooth loss in pike fish and 
displays the words five years 
old. This page is OT because 
it has keyword matches only 
and would be helpful for no 
users. 


OT SR R U V 


[mountain bikes], English 
(US) 

Find information about or 
purchase a mountain 
bike. 


Mountain Biking Tours 
mountainbiking.com.1 

A description for this result is not available. 


Links and ads only: Even 
though this LP has tabs and 
links that appear related to the 
query, neither the landing page 
nor the links provide any 
information about mountain 
bikes. The fake search page 
is useless and should be rated 
OT. 


OT SR R U V 


[how to quit smoking], 
English (US) 

Find information on ways 
to quit smoking. 


Benefit to quit smoking, alternative wavs to quit smoking, quit ... . .. 

users2.iabry.ccr " Benetit-to-quit-smoking htmi 

benefit to quit smoking i ivant stop smoking, quitting smoking chart, hypnotist quit 

smoking quit smoking feeling tired, free stuff to help stop smoking. 1 just quit ... 


Gibberish: This LP has 
gibberish text. Read this 
sentence: "The tropical 
ubiquity was being 
singularized burring." The 
quality of the landing page is 
so low that the page is OT. 


i^i^Kt ' 1 ' 1 ■ 1 ( • 1 ' 1 

or SR R y V 


[american express], 
English (US) 

Go to the American 
Express card or get 
information about the 
company and its 
products and services. 


Tlie Lipstick Chronicles: Identltv Theft Holiday Update 
thellpstlckchronlcles typepad com/ lipstick chronicles/ ./iden 
Nov 30. 2007 - The Lipstick Chronicles .... use it in a "chain" of infected machines to 
mount a brute force attack against, say. American Express or Wells Fargo. 


Insufficiently related to the 
query: The LP is a humorous 
blog post about a wife helping 
her husband buy a suit. The 
page mentions "American 
Express" on the page, but is 
insufficiently related to the 
query to be helpful for users 
and is OT for the user intent. 


OT SR R U V 
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Query and User Intent 



Result Block and LP, Utility Rating 



Description of the LP 



[every breath you take 
lyrics], English (US) 

Find the lyrics to the 
song "Every Breath You 
Take," which was written 
by Sting. 



Every Breath You Take lyrics - Free English Tests & Quiz for ESL/ . .. 

english-test,usj'--...'artist/.../1 41 1 97-every_breath jou_take-lyrics,hfm 

Every time that I come near her / 1 just lose my herve as I've dohe from the start / Every 

little thing she does Is magic / Everything she do just turns me on ... 



Insufficiently related to the 
query: This LP (on a low 
quality website with no website 
information and low quality 
text) displays the wrong lyrics. 
Even though the title on the 
page is "Every Breath You 
Take," the lyrics displayed are 
for the song "Every Little Thing 
She Does is Magic." 



[drive safely texas], 
English (US) 

There are several 
driver's education course 
websites with "drive 
safely" in the URL. 
Users may want to go to 
one of these websites. 



Safe Haven showtimes for Century Abilene 12 

1 hr 55mln - Rated PG-1 3 - Romance 

An afTirmlng and suspenseful story about a young woman's struggle to find 
love again after she arrives in a small North ... 
Century AOilene 12-3318 John Kno^ Drive. Abilene. TX - Map 
1 2:20 - 1 :40 - 3:00 - 4:20 - 5:40 - 7:00 - 3:20 - g:40pm 




Trailer | 



Insufficiently related to the 
query: This is a TLRB with 
special content. This result 
has no connection to the query 
and is OT. 



[what is wedding in 
Spanish], English (US) 

Find how to say the word 
"wedding" in Spanish. 



manana 

Translate "wedding" to Spanish 



This is a NTRB. The 
translation in the result block is 
completely wrong and 
inaccurate. 

Manana means "tomorrow," 
not "wedding." 



NASCAR Schedule 

Sprint Cup Series-Nationwide Serles-Camping World Truck Series 



[starting order for today's 
nascar race], English 
(US) 

Find information about 
the starting order of the 
Nascar race that day. 



# Date Race 


Winner/Time 

n 




- Feb 16 The Sprint Unlimited 


6:29 PM 


This is a NTRB. The content 


- Feb 21 Budweiser Duel 1 


11:18 


- Feb 21 Budweiser Duel 2 


1:00 PM 


is not what the user is looking 


1 Feb 24 Daytona 500 


10:30 


for and is OT. It does not 


2 Mar 3 Subway Fresh Fit 500 


12:14 


show the starting order of a 


+ Show more 


*AJI times are in FT 


Nascar event. 



OT SR R U V 
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Query and User Intent 


Result Block and LP, Utility Rating 


Description of the LP 




United Technologies Corporation 

NYSE: UTX - Feb 1 5 4:00pm ET 

90.78 +1.09 (1.22%) 






[united tech logo], 
English (US) 

Find images of the 
United Technologies 
Corporation logo. 


91.0 Open 90.05 

90.5 »/'Ai"-/*'*^'"Nr*v .>''%^'^^^j'"''^'*SA — ^'^"^ 

^^^f VnlLimp 4i?n4^1 

89.5 : ■ ^ AvgVal 4,071.000 

lOam 11 12 1 2 3 4pm uvlCsp 83 20B 

Id 5d tm em 5y ma« 




This IS a NTRB. The content 
is OT. The result block shows 
stock information, not the 
company logo. 


Google Finance -Yahoo Finance-MSN Money tfetSaipKr 








or SR R U V 






PERFECT COUPLE - Celebrltv CouDles Photo f2671 60081 - 


Fan... 


The LP on fanpop.com (with 
no IVIC other than images) has 
an image of Amanda Seyfried 
and Adam Brody, not Rachel 
IVIcAdams and Ryan Gosling. 

The MC of the page is not 
about Rachel McAdams and 
Ryan Gosling. There is a link 
to an image of them in the SC, 
but many users would never 
notice this link. This 
inconspicuous link does not 
provide enough utility to give 
the page a rating above OT. 


[rachel mcadams & ryan 
gosling couple image], 
English (US) 

Find an image of Rachel 
IVIcAdams and Ryan 
Gosling. 


vwftv fsnpop.com > ... > Celebrity Couples > Images > Photos 

PERFECT submitted by demidibi 2B71G008 ... Celebrity Couples PERFECT COUPLE 
More photos customize this Imagecreate cbiiage with Image PERFECT... 


OT SR R U V 




Louisville Zoo Membership Discount and Free Gift Card 

louisville.about. com/.. ./louisville-zoo-membership-cliscount-abcl -free... 

Sep 9', 2010 - Through October 4. new Louisville Zoo members can get a S5 ... 

Atlanta On The Cheap - Save Money Without Coupons at Zoo Atlanta ... 


Visit Zoo 


The LP on about.com has a 
short article about the 
Louisville Zoo. The page has 
a link in the SC titled "Visit Zoo 
Atlanta On The Cheap - Save 
Money Without Coupons at 
Zoo Atlanta." 

The MC of this page is not 
about Zoo Atlanta. The link to 
an article about Zoo Atlanta is 
in the SC and many users 
would never notice this link. 
This result should not be rated 
above OT. 


[zoo atlanta], English 
(US) 

Go to the Zoo Atlanta 
website. 


or R u V 
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15.0 Rating Porn (P), Foreign Language (F), and Didn't Load (D) Results 

You will assign P, F, and D flags to result blocks when appropriate. All flags are query-independent, meaning that they 
do not depend on the query. 

Here is a screenshot of the flags you will use when the result block has certain characteristics: 
BEE 



And here is a screenshot of a result block that shows when the P flag would be used. 



Query and User Intent 


Result Block, Flag, Utility Rating 


Explanation 


[freeones], English (US) 

Go to the Freeones 
website. 


Freeones: the Ultimate Supermodels - Celebs - Pornstars Link Site 
'A'W freeones.com/ 

Freeones: biggest resource to f nd porn stars and famous hot babes. 


The LP is the 
homepage of the 
Freeones website, a 
porn site. 

This result merits a 
Vital utility rating and 
the P flag. 


Q' E E 


1 1 ) 1 i| 1 i| ¥ 

OT 3R R U V 



15.1 Porn (P) Flag 

Please assign the Porn flag to all porn pages, whether the query is porn-seeking or not. The P flag does not depend 
on the query or user intent. 

A page will be considered porn if it has pornographic content, including porn images, links, text, pop-ups, and/or 
prominent porn ads. An image may be considered porn in one culture or country, but not another. Please use your 
judgment and knowledge of the locale. 

Remember: The P flag is used to indicate that the result is porn. It doesn't matter whether the query "asks for porn" 
or not. 



15.2 Porn Result Utility Rating 

15.2.1 Utility Rating for Clear Non-Porn Intent Queries 

If the user intent behind a query is clearly not porn-seeking, a landing page that has porn for its MC should be rated 
OT. 

• When the user intent is clearly not porn, a porn result should be considered unhelpful or useless. 

• Uninvited porn is a very bad experience for many users. 

Remember to assign the P flag for all porn pages. 
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Query 


User Intent 


Landing Page 


Utility 
Rating 


Porn Flag? 


r+rw/cl Pnnlioh {\ IQ^ 


rinu Toys 10 Duy. 


Example 

Warning - this page is porn. 




Vac 

Yes 


rhn\A/ tall iQ a pampll FnnliQh {\ 


Find tfie answer to tliis question 
about camels. 


Example 

Warning - this page is porn. 


OT 


1 Co 


[car pictures], English (US) 


Find pictures of cars. 


Example 

Warning - this page is porn. 


OT 


Yes 



Sometimes, the MC of a landing page is helpful for the query, but the page happens to display porn ads or porn links 
outside the MC. The query and the helpfulness of the MC have to be balanced with the user experience of the page. 
Use your judgment and represent users in your locale. 



15.2.2 Utility Rating for Possible Porn Intent Queries 

Some queries have both non-porn and porn interpretations. For example, all of the following English (US) queries are 
possible porn intent queries, t3ut they also have a non-porn intent: [girls], [gay], [thong], [breast], [sex], [spanking]. We 
will call these queries "possible porn intent" queries. 



For these queries, please rate as if the non-porn interpretation is dominant, even if you think users are looking for porn. 
For example, please rate as if a likely intent of [bikini], English (US) is shopping. Rate the porn interpretation as a 
minor interpretation, even if you think most users are looking for porn. 



Query 


User Intent 


Landing Page 


Utility 
Rating 


Porn flag? 


[breasts], English (US) 


Find anatomy or health 
information about breasts. 


Example 


U 


No 


[breasts], English (US) 


Find anatomy or health 
information about breasts. 


Example 

Warning - this page is porn. 


SR 


Yes 


[pictures of girls], English (US) 


Find pictures of girls. 


Example 


U 


No 


[pictures of girls], English (US) 


Find pictures of girls. 


Example 

Warning - this page is porn. 


SR 


Yes 



15.2.3 Utility Rating for Clear Porn Intent Queries 

For very clear porn queries where no other intent is possible, assign a rating to the porn landing page based on how 
helpful it is for the user. Even though there is porn intent, the page should still be assigned a P flag. 

Do not simply rate all porn pages for porn queries as R or U. Even though the query is porn and the result is porn, the 
page must fit the query and be helpful to get a high utility rating. 

Pages that provide a poor user experience, such as pages that try to download malicious software, should also receive 
low ratings, even if they have some images appropriate for the query. 

Porn stars, porn movies, names of specific porn websites, etc., can have V pages. Assign a P flag to all porn pages. 
Even Vital porn pages must be assigned a P flag. 
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Query 


User Intent 


Landing Page 


Utility 
Rating 


Porn Flag? 


[freeones], English 
(US) 


NavigaTe to ine rreeones nomepayG. 


Example: httoV/www.freeones.com 
Warning - this page is porn. 


V 


Vac 

Yes 


[freeones], English 
(US) 


iNavigaie to ine rreeones nomepage. 


Examoie: httD://www.baberoad.com 
Warning - this page is porn. 


OT 


Vac 

Yes 


Oenna Jameson], 
English (US) 


Find porn pictures of Jenna Jameson 
or go to her official website. 


Example 

Warning - this page is porn. 


w 
V 


Vac 

Yes 


Oenna Jameson], 
English (US) 


Find porn pictures of Jenna Jameson 
or go to her official website. 


Example 

Warning - this page is porn. 


nT 
\j 1 


Vac 

Yes 


[anime sex pictures], 
English (US) 


PinH aninriP qpy nir'tiirPQ 

nillU ClIIIIIIC oCA IJILfLUICo. 


Example 

Warning - this page is porn. 


Uto R 

range 




[cheerleader porn], 
English (US) 


Find porn pictures of cheerleaders. 


Example 

Warning - this page is porn. 


Uto R 

range 


Yes 



15.3 Reporting Illegal Images 
Child Pornography and Bestiality 

When working on rating projects in any task location or locale, you must follow United States federal law, which 
considers child pornography and bestiality to be illegal. 

Definition of Child Pornography 

An image is child pornography if it is a visual depiction of someone who appears to be a minor (i.e., under 18 years 
old) engaged in sexually explicit conduct (e.g., vaginal or anal intercourse, oral sex, bestiality or masturbation as well 
as lascivious depictions of the genitals), or sadistic or masochistic abuse. The image of sexually explicit conduct can 
involve a real child; a computer-generated, morphed, composite or otherwise altered image that appears to be a child 
(think of images that have been altered using "Photoshop"); or an adult who appears to be a child; and the image can 
be nonphotographic (e.g., drawings, cartoons, anime, paintings or sculptures) so long as the subject is engaging in 
sexually explicit conduct that is obscene. If it is indistinguishable from child pornography, it is child pornography. 

Even if the image has literary (think of the famous book "Lolita"), artistic, political (think of political cartoons), or 
scientific (think of images for a medical text book) value, please send the link to your vendor, as instructed below. 

Depiction of the genitals does not require the genitals to be uncovered. Thus, for example, a video of underage 
teenage girls dancing erotically, with multiple close-up shots of their covered genitals, or images of children with 
opaque underwear that focus on the genitalia could be considered child pornography. 

An image of a naked child (e.g., in the bathtub or at a nudist colony) is not considered child pornography as long as the 
child is not engaging in sexually explicit conduct, or the focus is not on the child's genitalia. Visual depictions of adults 
who look like children (e.g., a 35-year-old man play-acting in diapers, or an obvious woman dressed as a school girl) 
are not child pornography. If you don't think it's a minor, it probably isn't child pornography. However, if you cannot tell 
that the person in the image is over 1 8 (e.g., an under-developed 1 8-year-old whose body hair has been waxed), that 
is child pornography. 

Definition of Bestiality 

Bestiality or zoophilia is defined as human-animal sexual interaction. 
Reporting Instructions 

Please report illegal and offensive images as instructed by your vendor. 
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15.4 Foreign Language Results 

15.4.1 Using the Foreign Language (F) Flag 

Please assign a Foreign Language (F) flag when the language on the landing page is not one of the following: 

• The task language 

• English 

• A language which is commonly used by a significant percentage of the population in the task location 

For example, most users in Ukraine speak Russian. Therefore, landing pages in the Russian language should not be 
assigned the F flag for rating tasks in Ukrainian (UA). 



Here is a screenshot of a result block that shows when the F flag would be used. 



Query and User Intent 


Result Block and LP, Flag, Utility Rating 


Explanation 


[baidu], English (US) 

Go to the Baidu 
website. 


mw. baidu. com/ - Translate this page 


The LP is the homepage 
of the Baidu website. The 
result is the target of the 
query, but is in a foreign 
language (Chinese). 

This result merits a Vital 
utility rating and the F flag. 


B 0 E 


OT SR R J V 



Important: 



• Please assign the F flag even if you personally understand the language but most users in your locale do not. 

• Please remember to flag all foreign pages with the F flag, even if the query "asks" for a foreign language page. 

• Assign the F flag based on the language of the landing page, not the appearance of the result block. 

• Sometimes it is difficult to determine what language the landing page is in. The LP may have multiple 
languages or no words at all. In these cases, try to represent users in your locale. Does it feel like a foreign 
language page? You may look at IVIC, SC, Ads, and even the website the page is on. When in doubt, don't 
use the F flag. 



15.4.2 Utility Ratings for Foreign Language Results 

You must assign a utility rating for all result blocks in your task, even if the result blocks have a foreign language 
landing page. 

In most cases, pages you flag as F should be rated OT, because they cannot be understood by most users in your 
locale and are therefore useless. Remember that if users in your locale can read the language, then you shouldn't be 
using the F flag. Occasionally, you will encounter helpful F pages. 

If the query is clearly "asking" for a foreign language result, then the utility rating of the foreign language page should 
nof be OT. For example, please assign the V rating and F flag for baidu.com if the query is [baidu.com], English (US). 

Videos are often an example where foreign language pages are helpful and desired. Think about user intent and what 
pages are good for users. If the query "asks" for a foreign language song, band, film, sporting event, etc., then a video 
of the song, band, film, sporting event, etc. is helpful since it can probably be understood or enjoyed even though it is 
in a foreign language. For these types of queries, foreign language results are often expected. 

If the video is someone talking about the song, band, film, or event, the result probably cannot be understood or 
enjoyed and should be assigned an OT rating and the F flag. 
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Query and User Intent 


Result Block and LP, Utility Rating 


Explanation 


[gangnam style], English 
(US) 

VA/q+^K q \/iHo/~i /^r licton t/^ 
VVdloIl a VIUcU UI lloLc;il lU 

this song. 


Li 


PSY- GANGNAM STYLE MW- YouTube 


This is a very popular video in 
the U.S., even though the song 
is in Korean. This result is 
extremely helpful. The 
language of the LP is mostly 
English (even though the video 
is in Korean), so the F flag is 
not needed. 


ik to view th( 


lAww.y outube.com/watc h?v ==9b2k p7q 1 '^0 
1 Jul 15, 2012- Uploaded by offlclalpsy 

1 PSY - Gangnam Style (^J'liiSt'll) E> NOW available on ITunes; 
1 http:yySmarturl.it/psy9angnam [> Official ... 

3 video 


OT 


SR 


R U V 


[video of Celine dion 
singing s'il suffisait 
d'aimer], English (US) 

Watch a video of Celine 
Dion singing this song. 


Celine Dion - 


S'il Suffisait D'aimer Live - YouTube 


This video is just what English 
(US) users are looking for, 
even though the video is not in 
English. The language of the 
LP is mostly English (even 
though the video is in French), 
so the F flag is not needed. 


Link to view th( 


1 youtube.comftvatch7v=Hi.>;SIM09tl4 
1 Sep 2. 2011 - Uploaded by Jo]oMcLeod 

1 You need Adobe Flash Player to watch this video ... Celine Dion 
1 - S'il Suffisait D'aimer Live ... Justin ... 

3 video 


OT 


SR 


R U V 


[alex c], English (US) 

Watch a video or listen to 
a song by this German 
composer or find 

II IIUi M IciLIUi 1 ciUUUl llllll. 


Alex C ft. Yas 
Link to view th( 


3- Doktorspiele - YouTube 

■ wmv.yDutu be.cDm./watch?v=GKGmuBLd5VE 

1 f*lar 26. 201 1 - Uploaded by Thefjluzal 97 

1 Alex Christensen (ur. 7 tanrietnia 1 967 w Hamburgu) — muzyk 

1 niemiechti. DJ. znany tezjato Alex C ... 

3 video 


The query is for the German 
composer, Alex C. The 
landing page has a video with 
one of his songs sung by Y- 
Ass in German. It would be 
helpful for some or many 
English (US) users who type 
the query even though it is not 

III Cliyilbll. 1 IIU idiiyuciyc Ul Ulc; 

landing page isn't clear. Use 
your judgment when deciding 
whether to assign the F flag. 


or s! 


R U V 


[Kasal, Kasali, Kasalo], 
English (US) 

Watch a trailer of this 
Filipino film or find 
information about it. 


Kasal Kasalo Kasali - YouTube 

II I^MII "^P^ ^ 2DD7 - Uploaded by prevtrails 

H-r«t IotT^ Kasal Kasalo Kasali. prevtrails 4 videos. Subscribe Watch 

B- r ranm L^ter Kasal kasali, kasalo 0001 by... 

Link to view the video 


The query is for "Kasal, Kasali, 
Kasalo," a Filipino film. The 
LP is a clip from the movie in 
Filipino (Tagalog), the 
language spoken in the 
Philippines. It would be helpful 
for many English (US) users 
who type the query even 
though it is not in English. Use 
your judgment when deciding 
whether to assign the F flag. 


OT a? 


R U V 
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15.4.3 English Language Results 

The following rating guidance is for raters in non-English locales. You may stop reading this section if the 
language spoken in your locale is English! 

Your utility ratings need to reflect how helpful the result is for users in your locale. 

We know that you can read English (you are reading this document!), but you should only give high utility ratings to 
English results if users in your locale would expect or want them. Unless requested by the query, English results 
should be considered useless if most users in the locale can't read them. 



Here are some examples using Korean (KR) as the task language. In Korea, most users don't read English. Unless 
the query "asks for" English results, we will consider them unhelpful or even useless (OT): 



Query and User Intent 


Result Block and LP, Utility Rating 


Explanation 


[w]e5- o_Hj.Q^]^ Korean 
(KR) 


n|5f 2.uiDi - ^I^lu-H^K °EJ 3.^°\ 'i!^2^A^^^ - ^^|5|'iH2^ - Wikipedia 
ko.wilcipedia.org/wilci/ul ^_2Hhnh 

um eyoj 2H|.n|- 2H|(SCH: Baracic Hussein Obama Jr., Stm ui^3. a.u[n[, iseia 
81 4g ~ , DR tmo\ ^ S.MtW 1^=S)= DRel S*|OIHS ... 


This Korean language Wikipedia 
page on Barack Obama is very 
helpful for Korean (KR) users. 


OT 3R R U V 


lijja- o_4Q^ Is Barack 
Obama in Korean, so the 


Barack Obama - Wikipedia. tlie free encvclopedia 


This English Wikipedia page 
about Barack Obama has similar 
content to the Korean Wikipedia 
page. Although there are some 
helpful images on the page, very 
few users would be able to read 
this page. 

Note: this query does not "ask 
for" English language results. 


user intent is to find 
information about Barack 
Obama. 


en.wlklpecila.Qrg/wlKi/Barack_Obama 

Barack Hussein Obama li is the 44th and current President of the Uhited States. iHe is 
the frst African American to hoid the office. Born in iHonoiuiu. iHawaii. ... 


OT 3R R U V 


[Titanic 1997], Korean 
(KR) 

The user intent is to 
purchase a DVD or find 
information about the 
movie "Titanic," released 
in 1997. 


EfOIEfM - y|0|b1 g£f 

movie. naver.com/movie/bi/mi/basic.nhn?code... 

Al-olSEl robots.tj<t msoil S-SJjgshHl aSS A^St^ ^ KhUi^. m\t\ yo[s.J\ 


Although the query was typed in 
English, most Korean users 
would expect to see Korean 
language reviews or Korean 
stores for purchasing the movie. 

This Korean result on a Korean 
website is very helpful. 


OT SR R U V 


Titanic (19971 -IMDb 

m™.imdb.com/titie/tt01 20338/ 

Directed by James Cameron. With Leonardo DiCaprio. Kate Winsiet. Biiiy Zane. Kathy 
Bates A seventeen-year-oid aristocrat, expecting to be married to a rich ... 


This English language landing 
page has helpful content for 
users who can read English. 
Although there are some helpful 
images on the page, very few 
users would be able to read this 
page. 

Note: this query does not "ask 
for" English language results. 


or SR R U V 
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WWW. Samsung. CO m/sec/galaxytab/ 

y ^ A| g sg. 01 AHSS 1 Alej^l iJHifUP". GALAXY TAB 7.7 LTE 7:|-SSS 1 


Although the query was typed in 
English, most Korean users 
would expect to see the Korean 
page on the Samsung website. 






[Samsung Galaxy tablet], 
Korean (KR) 


OT SR R U V 


This Korean result is very helpful. 


Galaxy Tab: Android Tablet - Portable & Light Tablet 

www. s a m s u ng CO m.'u s.''mo bi 1 e.'g al a >; y-ta b.'' 

Meet the Samsung Galaxy Tab family including Galaxy Tab 10.1, 8.9 and 7.0 PIjs. 
Sort by size or carrier and find the Samsung Tab Android tablet that's right for ... 




The user intent is to 
purctiase, find 
information, or go to tine 
Galaxy tablet page on 
the Samsung website. 


This English page on the 
Samsung website, the company 
that makes this tablet, has 
helpful content. Although there 
are some helpful images on the 
page, very few users would be 
able to read this page. 


i^^^^^^v-l 1 t h 1 It 1 I 1 I t 




l^^^^v^l 1 f 1 [If 

OT SR R U V 


Note: this query does not "ask 
for" English language results. 



In some locales, English is one of the official languages or a commonly spoken language. In these locales, English 
websites are easy to use and could be helpful, depending on the query. 



For example, the Singapore government recognizes four official languages: English, Malay, Chinese, and Tamil, but 
English is the first and most dominant language in Singapore. 



Query and User Intent 


Result Block and LP, Utility Rating 


Explanation 


[Barack Obama], 
Chinese Simplified 
(SG) 

Find information about 
Barack Obama. 


Barack Obama - Wihtipedia. the free encvclopedia 

en ',v - :t :■ j ■.iKi/Barack_Obama 

Barac;l< Hussein Obama II is the 44th and current President of the United States. He is 
the frst African American to hold the offce. Born in Honolulu, Hawaii, ... 


This Wikipedia page in English 
about Obama would be very 
helpful to users in Singapore. 


OT SR R U V 


B+^siR.jSLRBi-aKm, ^ fiRff^m^-^^ -s*m-wikiDedia 

2h.wil;ipedia.org.i'zh-cn..'m^iS II 

EtsS-tlSB-SE-^ZtS: (sSif : Barack Hussein Obama II. ,'ba ro k hu sein oo 
bams.': 19B1^6H4H-) . «HR±SigiEi[;(j0 . |i441f HaS.iS . M ... 


This Wikipedia page in 
Chinese about Obama would 
also be very helpful to users in 
Singapore. 


-f -r- f ; -f-r- f -,'-f-v , | , | 
or SR R LF V 



15.5 Rating Results with Content that Doesn't Load 
15.5.1 Using the Didn't Load (D) Flag 

Didn't Load (D) is used to indicate technical problems that prevent you from viewing any LP content. 
Use the D flag when: 

• The MC of the landing page is a web server or web application error message and there is no other content 
on the page (no navigation links, no home link, no SC, no Ads). See here for a Wikipedia page on different 
types of error messages. 

• The landing page is completely blank (no MC, no SC, no Ads). 
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Assign the D flag based on the landing page, not the result block. 

Here is an example of a D landing page. You cannot tell that the landing page doesn't load by looking at the result 
block. 



r^iiofir £>nH 1 leaf Infant 
VjiUcry diiu User iiiiciil 


Result Block and LP, Flag, Utility Rating 


F-vnIanfktinn 
CApidlldllUll 


[douglas instruments], 
English (US) 

Navigate to the 
homepage of this 
website. 


Douglas Instruments Success in Protein Crvstaiiization 

\nvx . d 0 u g 1 a s . c 0 . u to' 

Douglas Instruments designs and manufactures robots for protein crystaiiization. Our 
systems are setting new standards in protein crystaiiization success ... 


This page displays a generic 
404 message. There is no 
MC, SC, or Ads on the page. 

This result merits an OT utility 
rating and the D flag. 


B E m 


OT 3R R U V 



Here are screenshots of other types of landing pages that should be assigned the D flag: Dl, D2, D3 . 
Additionally, use the D flag for 

• Malware warnings, such as "Warning - visiting this web site may harm your computer!" 

• Pages that solicit certificate acceptance requests. 

Here is an example of a malware warning: 



Warning; Visiting this site may harm your connputtrl 

Tt* W-S»Jr y. woi v.«p|H()pLtnni >.iTi»^-t tirtrif laiiil-M ^1^ ptJPghi fliiwivHi.cpm i^^Mtf 
nitwit* - idmtM thit CM IM ytaJ HfflfUlW OlhH^iM 4^inl* tfttbii ytU CHHM Jtrti ■ 401; IM eB^BM 

Liainiiaii,ilinJI»«1ii»iminmmn»Ho<n<j«^»J«»ni^ 



Here is an example of a certificate acceptance request: 



} WeJj^iLc Ctrtllifi'd by dti Uriknawn Autliaiitv 



it 



PussMe reasons, for this emor 

- Your hrgwssr ckm net reco^nizs Vtm C4rtifi(jlt* AiJthDiity that it^ttwd th« 
site's certificate.. 

' The s,ite's {ertD^iEate is incomplete due Es a seiveF mlsunriguratian. 

- You are cannected to a sile pf^tanding ba tie ', possibiy tD abtain your 
ca<iifidentiai infomriaticin. 

Pieis« fiabfy Ehe utit's, •utiuniis.tet abenA this pmhltm. 

fi€f!TO accepting tNs certificite, you shouFd «j(3rnine this site's certificite 
car«fulty. Are yau wiiiing tio to accept ihis certifJcabe for the purpcse of 
identtfy irtg the Web site 'T 

Examine OrtrTicate.. , 

Accept thi^ certiiifLate permane^iy 
* Accept this rertiricste- temporarily Fw this jessiijni 
aa not accept this certificate and do not connect to this Web site 

cantal ' | OK 
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15.5.2 Rating Results that Do Not Load 



All result blocks must be given a utility rating. If the landing page truly doesn't load, assign the D flag and rate the 
page OT. True Didn't Load (D) pages are useless. 



Sometimes the page partially loads or has an error message. Give utility ratings based on how helpful the result is for 
the query. Error messages can be customized by the webmaster and are part of a well-functioning website. 
Sometimes these pages are helpful for the query. 



Query and User Intent 


Result Block and LP, Utility Rating 


Explanation 


[boys snow shoes], 
English (US) 

Find information about or 
purchase boys' snow 
shoes. 


Grizzlv - EXOSHOP 

e.>:osh op.c om/en/skatesh op/g ri ptape/g rizzly-sta m p-bl u e/ 

Bearings ■ Bushings ■ Compietes ■ Decks ■ DVD ■ Griptape ■ Hardware / Risers ■ 

Longboards ■ Magazine ■ Misceiianeous ■ Protective Gears ■ Snow sHate ■ Toois ... 


The IVIC has an error 
message, but the LP has a lot 
of SC. However, the page has 
no information about boys' 
snow shoes and is unhelpful 
for the query. 

Do not assign the D flag. 


OT SR R U V 


[bible passages], English 
(US) 

Find specific passages in 
the bilDle. 


BibleGateway.com: A searchable online Bible in over 100 version... 
i«»v. b 1 b 1 e gateway. CO m / 

We've just added three new Bible versions to our emaii Verse of the Day! You can row 
add the E.>(panded Bible. New Century Version, and New King James ... 


In spite of the customized "No 
results found" message on the 
LP, it has links to all passages 
in the bible, organized by 
book. It would be very helpful 
for most users. 

Do not assign the D flag. 


OT SR R y V 



16.0 Location and Utility Rating 

16.1 Locale is Important 

in Section 12.0 on Understanding the Query, we gave an example of a query that means different things to users in 
different locations. It is important to think about the locale when thinking about the query. 




Query: [football] Query: [football] 

Locale: English (UK) Locale: English (US) 
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You must also think about the location of the results you are rating. For some queries, results from other countries 
may be helpful or desired. 

For other queries, results from outside the rating locale may be unhelpful or useless, even if the result is in the right 
language. For example, most users want to make purchases from stores in their own country, unless the item is very 
unusual. 



amazon 




co.uk YotirAmazon CO uk Toda/sDeals GittCards Help 
search Fi^aw ▼ 



The Amazing Spider-Man DVD [2012] 

£8.50 

Delivered FREE in the UK 

In stock 

Dispatched from and sold by Amazon.co.ul(. 
Gift-wrap available. 




Not helpful 



Query: [Spiderman DVD] 
Locale: English (US) 



™ 1 ^ 



Vojf Amazon.com ToOay's Deals Gin Cards Help 
Search Fiimsw^ 

The Amazing Spider-Man DVD [2012] 
$16.09 

Delivered FREE in the United States 

In stock 

Dispatched from and sold by Amazon.com. 

Gift-wrap available. 



Helpful 



Do not hesitate to lower the rating to OT if there is a mismatch between the locale and the result which makes the 
result useless for a user in the locale. 



Query and User Intent 


Result Block and LP, Utility Rating 


Explanation 




White Chocolate Blueberry Cheesecake Recipe - Aiirecipes.com 

^ aiirecipes.com/ .j'white-chocolate-blueberry-.-- 
tST] ' " ifk-k-k-i Rating: 4.7 - 61 reviews - 1 hr 20 mins - 603 cai 

What coLid be better? White chocoiate, cheesecaise and blueberry 
^^^^^ topping! 


This result is helpful for 
users in the U.S. The 
ingredients and 
measurements are familiar 


[white chocolate berry 
cheesecake recipe], 
English (US) 


or SR R U V 


to U.S. users. 


Find a specific type of 
cheesecake recipe. 


White chocolate berry cheesecake recipe - Recipes - BBC Good... 
^**dftfl mw.tabcgoodfood. com/recipes/.. ./white-cho... 
* r i^"^ k-k-k-k-k Rating: 5-79 reviews 

' •j£_^.^ Jun 10.2009 - Breal^ the chocolate into a glass bowi, then put it over a 
^00^^ pan of just simmering water (d meit. making sure the bottom of the bowl 
doesn't touch ... 


Most U.S. users would 
have difficulty making this 
recipe. The measurements 
are in metrics and some of 
the ingredients and 
terminology are British. 








OT SR R u v 


This result is not helpful for 
U.S. users. 
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Query and User Intent 


Result Block and LP, Utility Rating 


Explanation 


[human rights violations], 
English (US) 

Find examples or 
information about human 
rights violations. 


Iran - Human RIahts Watch 

www. h ™. arg/m i d d 1 e-ea stn -af ri c a/i ra n 

Any criminal investigation against Saeed Mortazavi. the former Tehran Prosecutor 
Generai, shouid inciude the serious human rights abuses of which he has ... 


Most people in the U.S. would 
be interested in international 
human rights violations. For 
this query, results about 
countries other than the U.S. 
are fine. International news 
sources might be some of the 
best resources. 

All these results are helpful 
for the query. 

Use your common sense to 
decide what a user in your 
location would be interested 
in. 


OT SR R J V 


Prison and Detention Conditions - Human Rights Watch 

wx'ti.Ttm arg.'unlted-statesf'us Cprison-and-detention-conditio 

Such failures violate the human rights of all persons deprived of their liberty to be 

treated with ... USyNew Yorki Few Arrested for Pot Become Violent Criminals ... 


OT SR R J V 


Human rights in Sudan - Wikipedia. the free encyclopedia 

en.wikipedis.org/wiki/Human_righTsjn_Sudan 

Jump to Prisoner abuse: Several hundred adults and children were imprisoned after 
members of the Justice and Equality Movement attacked Khartoum in ... 


OT SR R J V 


[washing machines to 
buy], English (US) 

Buy a washing machine 
or compare prices on 
washing machines. 


Washing machines | Compare prices on washer dn^ers - Kelkoo 

mw.kelkoo-CO.uk > ... > Laundry & Cleaning 

Save money and time buying a washing machine by comparing prices and features 
using our price comparison pages. It's a quick and easy way to find the Pest ... 


Most users want to purchase 
common items from stores in 
their locale. 

It would be too expensive to 
ship a washing machine from 
the U.K. to the U.S. This 
result is useless for U.S. 
users. 


OT SR R LJ V 


[house painters], English 
(US) 

Find a company to do 
house painting. 


Painting Contractor Liverpool. Mersevside 

wn.w. c h 1 d 1 owd ec orator s. c 0. u It' 

chidiows painting amd decorating contractors, painters decorstors.painters liverpool. 
painters manchester. painters anglesey. painters blackpool. painters ... 


A painting contractor in the 
U.K. is useless for a U.S. 
user looking for a house 
painter. 


OT SR R U V 


[car insurance], English 
(US) 

rlUbcaloiI Ul puioilabc 

car insurance, or 
compare car insurance 
rates. 


Car insurance Ireland - Tesco Car Insurance 

m™.tesco.ie;finance,xarinsurance.'default-htm 

Get a quote for Car Insurance in ireiand, quaiity cover. Tesco price, see how much you 
could save wrth Tesco Car Insurance, j Tesco Car Insurance. 


An insurance company that 
operates in Ireland and sells 
insurance to users in Ireland 
is useless for U.S. users 
researching car insurance. 


OT SR R u v 


[purchase boys bedding 
online], English (US) 

Purchase bedding for a 
boy's room online. 


BOYS ROOM - Cottonbox 
WiV.cottontaox.com.au/catalog. aspx?departmentid=2 

Ahoy Quiit Cover Set by Cotton House Kids 20% off from SI 27.95. Android Quilt Cover 
Set by Whimsy. Android Quilt Cover Set by Whimsy 25% off from $97.45 ... 


Cottonbox is a children's linen 
store in Australia. They ship 
to Australia and New 
z.eaiana, so this result is 
useless for U.S. users looking 
to buy linens. 


T 1 1 1 1 1— ^u— -i^«^ 

OT SR R U V 
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16.2 Queries that "Ask" for Nearby Results or Nearby Information 

Some queries "ask" for nearby information or nearby results. We call these "local intent" queries. 




F.ns< J 
oak SI 






In* f^'^^^ 
o 

0. 

Q page ^' 



F«»5 



Local Intent 



— jame&H 

Leng FieW * 



30 / 86 



OahSl 



Local Intent 



I want to get Chinese 
food for dinner. 




Query: [Chinese restaurant] 
Locale: English (US) 





Query: [weather] 
Locale: English (US) 



Here are more examples of local intent queries: 
[pizza] 
[time] 
[weather] 
[yoga class] 
[coffee shops] 
[movie showtimes] 
[train station] 
[car repair] 
[plumber] 
[swimming pool] 
[dentists] 
[farmers market] 
[bank of america atm locations] 
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Many queries are clearly not asking for nearby information or nearby results. We call these "non-local intent" queries. 




Web Images Maps Shopping News- Moia ' Searcti IodIs 

About 70.600, QOO results ID 30 secords) 

IQTe*tcom 

wj'w iqtesf com' 

Fiee lest wiifi infomaligr gr jnlelltgence testtng and history Wereof Additional 
deleiled reports ^re avsljable tor a tee 

Two Caulions - Wliat is sn 10 Score? - Insinictions tor Itie Irlelligence - Login 
LQ TgatLaha^fKejJitlmie.leslinq 

winv iiifeili'gercetesl: com/ 

Oniina ftaa Iq tast - take this 10 rriirLti qui? and get ycur score and testing results 
sutamaticaily 

Intelligence quotient - Wikioedia free encyclopedia 
ei v^iKipedra org-'wiki 'I nielli gencie_quoliern 

Jump (0 lilodem 1Mb Well-kiown modeii IQ tests mclude Raven's Pfogressive 
Matrices, Wechsler Adult lirtelligence Scale. Wechsler Irlelhgence Scale ... 

Free IQ Test - Fast. Free and Accurate Online IQ Test 

AW lise-iqtest nsli' 

Take ojrfasi fiee and accurate online 10 test to gel ycor IQ score quickly and find out 
how snait ycu are free of charge 

£,^£5 people +1"d this 



I would like to take an 
IQ test online. 




Query: [iq test] 
Locale: English (US) 



Non-Local Intent 



Here are more examples of non-local intent queries: 

• [hotmail] 

• [easter] 

• [scrabble cheat] 

• [translate] 

• [definition of sedentary] 

• [aapl] 

• [beyonce] 

• [angry birds] 

• [small dog breeds] 

• [dance videos] 

• [oscars2012] 

• [pick up lines] 

• [boyfriend lyrics] 

• [bank of america login] 
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And some queries could go either way. Some users may want nearby results and others may not. 




Walmart:, 




Hnvlrv Hn-A A Honlit 


^ There's still time to treat her 


Aivarrl, Shiwi.& Ivwalrv 




Antv « M«m Jnwrtmnml 

Unv Lrjn ■ PariT ^upalkn 
Ptumucv- HHiHh ftpMiEv 


^^^^ ^ 






Sm Ml Horn 

LrWMt-TdW ^MK>« Dri 
Canon C>n«ru, (rwn i a 




-1 o»»g)l«."<-?? 



Non-Local Intent 




Here are some examples of queries with both local and non-local intent 

• [hotels] 

• [post office] 

• [apple store] 

• [citibank] 

• [bmw] 

• [best buy] 

• [office depot] 

• [target] 

• [library] 

• [bank of america] 

• [the gap] 

• [barnes and noble] 

Use your common sense when thinking about queries and whether they have possible local intent. 
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16.3 User Location and Understanding the Query 

Every query has a locale. Some queries also have information about where the user is located: the user location. The 
user location is always inside the locale and is usually a city. 

Sometimes the user location can change our interpretation of the query or our understanding of user intent. 

For users in Sunnyvale, California, the query [turmeric] could have two different interpretations: a popular restaurant 
named Turmeric or the spice turmeric. 




Local Intent 



' a 



create account A Leg in 



WlKIPEDiA 

Mair cage 
contents 
Feaiurea canteni 
Current events 
Randflin slide 
OanaltloWiMp^dla 

Inleraction 

Help 

Community E'Qrtal 
Kecetil Changes 
CflfiiadWiKipgdia 

t- Toolttcn: 



Re.ad Edit Viewhistoiy jSesfcli 



Turmeric 



From .Vif^ipsda ttiefreeencydDpeda 

Tuimeric {Carcuma Jonga) Engbh intrnunciatiDn / 

\^ msnkl \i a ihaaimaaus hsfbscsou^ p^t^nmsl p\anr ot 
Wie ginger family. Zingiberaceae.'^' tt is nalive 1o tropical 
Sojlh Asis snd needs temperatures tselween 20°C and 
30°C [6S'F and 85°F) snd m considerable amount of 
anr^ual rainfall to ilirr.'e Plants are gathered annually for 
tiieir rhi;4)mes. and propagated horn some of tlir>se 
rtiizomes in folloviirg season 
When nnl us^d fresh Ihe iiizomes are bailed ior several 
liDurs snd then dried in tist ovsm. aFter kvhicti they are 
ground into s deep orange-yellon powder commoni/ used 
as a spice in curries and other South Asian and Middle 
Easiam cui&ins. for dy^irrg. and id impart color lo 
rruaSard condinients Its aclive ingredient is curcumir 



Indian Turmeric 




Non-Local Intent 



I want to learn about 
the spice turmeric. 




O 



O 



Query: [turmeric] 

Locale: English (US) 

User Location: Sunnyvale, CA 
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In most other user locations, there is no restaurant (or anything else) named Turmeric. In most user locations, there is 
just one interpretation of the query [turmeric]: the spice. Even with no user location, we will assume that the dominant 
interpretation of [turmeric] is the spice. The Sunnyvale restaurant is not well known outside of Sunnyvale, California. 




Create account ^[j03\a 



WlKIPEDiA 



Main page 
Contents 
Featured corlenl 
Cunent ^@nts 
Random slide 
Donate to^Vlkiciedla 

- I rile faction 
Help 

About uvmpedia 
Community poital 
Recent disnges 
Carrlaawndpfrda 

^ TOOli&OK 



Rsad Edit Visw history |Sea-cli 



Turmeric 



FromWikipedIa, theltee ercydopeiia 

Tumeric {CarcumB longa) Englin uronunciiMrj / 

mank/ is a rliizoiratous herbaceous perennial plant cf 
the ginge' faniily, ^ingibersce^s It is native to tropical 
South Asia and needs tempeiatures between 20°C and 
30°C (S3T and 8GT) and a conslderabJe amount of 
annual rain^ I to thfive Plants are gathered annually for 
their rhizomes, and propjag^s-J frgmMme of those 
[fiizomes in the following reason 
When not jsed fresh the rhizomes aie boiled for several 
haurs and then dned in hot ovens, al^er whicH they are 
ground into a deep orange-yellow powdei commonly used 
as a spice in crimes and olhei South Asian and Middle 
Easiem cuisine, for dyeing, and lo impart color to 
mustard condiments. Hs aciwe Ingredient is curcumin 




I want to learn about 
the spice turmeric. 




Query: [turmeric] 
Locale: English (US) 
User Location: Lincoln, NE 



Non-Local Intent 



For many or most queries, the user location does not change our understanding of the query. Here are some 
examples: 

• [facebook] 

• [pictures of kittens] 

• [how many miles between the earth and the moon] 

• [reviews of hd tv] 

• [yahoo mail] 

• [Chinese restaurants] 

• [pizza] 

• [citibank] 



When is the user location important in understanding query interpretation and user intent? Sometimes or rarely, it 
depends on the query. Please use both web research and your personal judgment to answer this question. Ask 
yourself, "Would users in one city be looking for something different than users in another city?" For many queries, the 
answer to that question is "no." 



Proprietary and Confidential - Copyright 2014 



122 



16.4 Queries with an Explicit Location 

Sometimes users tell search engines exactly what kinds of results they are looking for by adding the desired location 
the query. We'll call this location inside the query the "explicit location." The explicit location makes queries much 
easier to understand and interpret. 



Web laagii Maps Snapping Uoi? • StiF^h Idols 
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--fa 



Wa hM (mc* nutcli guirM««[ 
Otf In HoiDli 

«»S.ira jwople +rd3rtDlloM 
tWdxcoir 

ThfeRjC-CarHon^nai 

□us arrtl UnMnd i/Kf Champagne 
«. S^D Cr«lll u Spi Fram Jd6»j'iil 
rii.i 21Z-1 t^cKinnry Annin Qillai T. 



My mom is coming to 
visit. I need to find a 
fiotel for her nearby. 




Query: [Dallas hotels] 
Locale: English (US) 
User Location: Dallas, TX 
Explicit Location: Dallas 



N ew YorK HottW - Em«tia ^ ^^^^ 
t;i)«a .\ hAi «[ ih NmYoix? ChoH* ttwti m 

^Vfl»lff.■^l ysiu ^ifOgvl. compir* pie»» ■ 
NiM Ytirk-MMW yark Hold A rjS.Uft (La* V*0>t lliHliW Starts «l ... 



pr *ll piji M#w ,r. 



1 Mfliail, 



Hew Yuft aiy Hi*eh an TnpMnEei Find SeUSSi tiwelei rmsHS d?BS7 cbihM 
FticliDS srcl prices (H433lKitEli irNawf Tort Cny. HBwrDFH 
Ci-sablanta Hn1«irirT>es SquHG - HoieIb - S[ay Hfh Vnik [^ky Rcscaiirdrls 

Nflw Yofh ■ Nwv Yorfc Holal and Casirw (Ui» Ve^a*. mi Resort 

*♦+♦ . fijim^ d . Ji07 rsvitws - Prrc* ran^ S 
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4 

Map for r*w yo-rk hcteis 
Hoteiim Haw York ^ ^ 

Offoal sue ■ SSimooi tfauU 

New York Hotela 

SsM upta5a% sn yaur Houl 
Limilad Tms Only - Dvrl Mas Oii! 
JM D2j ffcrte 4l'dorli>llr>«0rhl; 

HoldiiiriHwYoihCilt 

F(i« HoralDialaEaiilr 
CofUfin Oti\t Fro«> lOCK Of Sitn 



I need a hotel for my trip to 
New York. 




Query: [New York hotels] 
Locale: English (US) 
User Location: Dallas, TX 
Explicit Location: New York 



Sometimes the explicit location matches the user location or locale, and sometimes it doesn't. 



When there is an explicit location in the query, pay attention to it! Users have told the search engine something very 
important by including exactly what they are looking for. 
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16.5 Vital Ratings for Queries with a User Location 

In most cases, the user location does not affect the Vital rating. 



Sometimes, user location allows us to better understand the intent of the query and give a result a Vital rating. Here is 
an example of this relatively rare situation. There are multiple libraries named Belmont Library in the U.S. Without a 
user location, we do not know which library the user is looking for and no V rating is possible. With a user location, a V 
rating is possible. 




Belmont Massachusetts Library 




Belmont California Library 



Proprietary and Confidential - Copyright 2014 



124 



16.6 Rating Queries witli User Location and Explicit Location 
The user location may play a large role in assigning utility ratings when: 

• The user location changes our understanding of the query or user intent. 

• The query is a local intent query, i.e., the user is looking for local information or results nearby. 

For many queries, the user location is not very important when assigning utility ratings. 

The explicit location always plays a large role in understanding the query and assigning utility ratings. 



16.7 Rating Local Intent Queries 

When there is a user location for a local intent query, such as [Chinese restaurants] with a user location of Boston, 
MA, results in or near the user location are the most helpful. 

How close is "near"? Most people are not willing to travel very far for a gas station, coffee shop, supermarket, etc. 
Those are types of businesses that most users expect to find very nearby. 

Users might be willing to travel a little farther for certain kinds of local results: doctors' offices, libraries, specific types 
of restaurants, public facilities like swimming pools, hiking trails in open spaces, etc. Sometimes users may accept 
results that are even farther away, such as a very specialized medical clinic. 

In other words, when we say users are looking for results "nearby," the word "nearby" can mean different distances for 
different queries. As always, please use your judgment. 



16.8 Rating Examples with User Location, Explicit Location, and Local Intent 



Query, Explicit and User 
Locations, User Intent 


Result Block and LP, Utility Rating 


Explanation 


Query: [Chinese 

restaurants], 

English (US) 

Explicit Location: None 

User Location: Mountain 

View, California 

User Intent: This query 
has local intent. The user 
wants to find Chinese 
restaurants in the city of 
l\/lountain View, California. 


Best Chinese restaurant Mountain View. CA - Yelp 

www. yelp. com/search?.. .Chinese+Restaurant...Mountain+ View... 
Reviews on Best Chinese restaurant in Mountain View, CAKirin Chinese 
Restaurant, Fu Lam Mum, Chef Xiu, Chef Liu Mandarin Cuisine, Queen House. ... 


This LP is a fairly 
comprehensive list of 
Chinese restaurants in 
IVIountain View. Many 
restaurants on the list 
have reviews. There is a 
map and features to 
narrow down the 
restaurants to certain 
categories. 


OT 3R R U V 


Fu Lam Mum: Dim Sum Restaurant in IVfountain View. CA 




fulammum.ccrtv 

Fu Lam Mum Offers Free Online Ordering ForTal<eout. Checl< Out Our Locai Menu For 
Mountain View, CA. 


This landing page is the 
website of Fu Lam Mum, a 
well-l<nown Chinese 
restaurant in the user 
location. 


OT 3R R U V 


Dragonfly Mandarin :: Voted Best Chinese Restaurant in Chicago 
wmv.dragonflymandarin.com/ 

Twitter button Facebool< button. (312) 455-1400. info@dragonflymandarin.com. 832 W. 
Randolph St.. Chicago. iL 60607. ©2004 - 2011 Dragonfly Mandarin. 


This landing page is the 
website Chinese 
restaurant in Chicago. It 
is unlikely users in 
Mountain View would find 
this helpful. 


OT SR R U V 
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Query, Explicit and User 
Locations, User Intent 


Result Block and LP, Utility Rating 


Explanation 




Maldonado's Pizza - Mountain View. CA 

mnv.malclonacIospizza.com/ 

Come to Maldonado's Pizzeria, a famiiy-friendiy piace offering the best homemade, 
mouth watering hand-tossed pizza this side of New Yorl<! We aiso teature a ... 


This is a very popular 
pizza place in IVIountain 
View, CA. 




OT SR R U V 




Query: [pizza], English 
(US) 

Explicit Location: None 
User Location: IVIountain 
View, California 


New York Pizza | IVIountain View, CA 94040 | Order iPizza Online ... 

n ei,vyorl<pi iza m ou nta i nvi ew. c om.;' 

View the menu, reviews, and coupons for New Yorl< Pizza iocated at 1 040 Grant Rd in 
Mountain View. CA 94040. Order New York Pizza deiivery oniine or by ... 


This is one of many pizza 
places in IVIountain View, 
CA. 

This website is "powered 


or SR R U V 


ijy iviyr izza.com ana 
there is not a lot of 
website information for 
New York Pizza. 


User Intent: IVIost users 
want to find a pizza place 
nearby. 

Tfiis query lias clear local 
intent, whether or not the 


9th Street Pizza 

wrx. 9thstreetpizza.com.i'' 

Come in and try the best New Yorl< pizza in town! Our deiivery service is FREE! ... 231 E 
3th Street Los Angeles. CA 90015 213-627-7798 / Fa.>: 213-627-2697 ... 


This is a pizza place in 
Los Angeles, which is 
about 350 miles from 
IVIountain View, CA. 


task has a user location. 


OT SR R U V 


This result is unhelpful for 
this query. 




Pizza - Wikipedia, the free encvclopedia 

en .wi l<i ped i a - org.i'wi ki.'Pi zza 

Pizza is an oven-baked, fat. round bread typicaiiy topped with a tomato sauce, cheese 
and various toppings. Pizza was originaiiy invented in Napies, itaiy, and ... 


IVIost users are looking for 
a pizza place nearby. 

The LP has a lot of high 
quality IVIC, but finding 
pizza information is a less 
likely user intent. 




OT 3R R U V 
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Query, Explicit and User 
Locations, User Intent 


Result Block and LP, Utility Rating 


Explanation 


Query: [target], English 
(US) 

Explicit Location: None 
User Location: None 

User Intent: This query 
hr59 both local and non- 
local intent. Users may 
want to shop online or visit 
a store nearby. 


Target.com : Furniture, Babv, Electronics, Tovs 

target.com,' 

Expect more pay less with Target. Spend $50, get free shipping on over 500K items. 
Choose from a wide selection of furniture, baby, electronics, toys, shoes, c. 


The dominant 
interpretation of this query 
is the Target store and 
this is the official Target 
website. 


or SR R U V 


Query: [target], English 
(US) 

Explicit Location: None 
User Location: Keizer, 
Oregon 

User Intent: This query 
has both local and non- 
local intent. Users may 
want to shop online or visit 
a store nearby. 


Target.com : Furniture, Babv, Electronics, Toys 

mvA'.targetcom./ 

Expect more pay less with Target. Spend $50, get free shipping or over 500K items. 
Choose frorr a wide selection of furniture, baby, electronics, toys, shoes, c. 


The dominant 
interpretation of this query 
is the Target store and 
this is the official Target 
website. 

The LP of target.com is V 
for this query even when 
the rating task has a user 
location. 


OT SR R U V 


Keizer Now with Fresh Grocery - Target 

rg et . c om/sto re . . ..I'sto re -d eta i 1 s? sto re N u m ber. . . St 0 re D eta i 1 s. . . 
Keizer Now with Fresh Grocery. 6450 Keizer Station Blvd NE Keizer, OR 97303 (503) 
856-0601. Weekly Ad ■ Get Coupons. Get Directions ... 


This LP has the address, 
phone number, and hours 
for the Target store in 
Kpi7pr Orpnnn Thi^ 
result is very helpful for 
users who want to visit the 
store. 


OT SR R U V 


Query: [target keizer], 
English (US) 

Explicit Location: Keizer 
User Location: None 

User Intent: This query 
has local intent. The 
explicit location (keizer) 
tells us that users are 
looking for the Keizer 
location of Target. 


Keizer Now with Fresh Grocery - Target 

mvw.ta rget.com/store.../store-details?storeN umber.. .StoreDetails... 

Keizer Now with Fresh Grocery. 6450 Keizer Station Blvd NE Keizer, OR 97303 (503) 

856-0601. Weekly Ad ■ Get Coupons. Get Directions ... 


The explicit location 
(Keizer) tells us that users 
are looking for the Keizer 
location of Target. This 
LP is the official page for 
the Target store in Keizer, 
Oregon. 


OT SR R U V 


Query: [target keizer], 
English (US) 

Explicit Location: Keizer 
User Location: None 

User Intent: This query 
has local intent. The 
explicit location (Keizer) 

tollc lie that 1 icorc 3ro 

looking for the Keizer 
location of Target. 


Target.com : Furniture, Baby, Electronics, Toys 

Expect more pay less with Target. Spend $50. get free shipping on over 500K items. 
Choose frorr a wide selection of furniture, baby, electronics, toys, shoes, c. 


The explicit location 
(Keizer) tells us that users 
are looking for the Keizer 
location of Target. This 
LP is the Target 
homepage. Although 
there is no information 
about the Keizer store on 
the LP, there is a "find a 

ctnro" link VA/hir'h iicorc 

oLUIc; llllt\ Vvl IILrl 1 Uoc^io 

would expect to find. This 
result would be helpful for 
many or most users. 


1 1 1 1 1 1 1 1 

OT SR R U V 
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Query, Explicit and User 
Locations, User Intent 



Result Block and LP, Utility Rating 



Explanation 



Minimum wage - Kansas Department of Labor 

wifx. d ol . ks. g ov/Laws/F AQ wa g e s . a spK 

What is the minimum wage in Kansas? The [Kansas minimum wage is S7,25 per hour. 
What is the ... Your em pi oyer can not change your rate of pay retroactiveiy. 



Query: [minimum wage 
rate], English (US) 
Explicit Location: None 
User Location: Wicliita, 
Kansas 

User Intent: Find out 
wtiat tlie minimum wage 
is. Different states in the 
U.S. have different 
minimum wage rates. 
Users might be interested 
in the minimum wage in 
their state and how it 
compares to other places. 



OT 



This is the official Kansas 
state website page on 
minimum wage for 
Kansas. 



Minimum Wage Laws in the States - Wage and Hour Division (WH . 

mn\i , d oi . g ov/wh d/m i n wa g e/a m eri c a . htrr 

Minimum wage rate and overtime provisions appiicabie to retaii and service, 
commerciai support service, food and beverage, and heaith and medical industries. 



I • 

OT 



This LP has the minimum 
wage rates for all states. 
It is on the official U.S. 
Department of Labor 
website. 



Minimum wage 

vwA'. d i r. c a . g ovi'd i se./f a q_m 1 n i m u m wa g e . htm 

Thus, since Caiifornia's current iaw requires a Inigher minimum wage rate than does the 
federai iaw, aii empioyers ir Caiifornia who are subject to both laws ... 



This is the official 
California state website 
page on minimum wage 
for California. 



or SH 



Images of pictures of l<ittens 

bing:.com/images 



Query: [pictures of 

kittens] 

English (US) 

Explicit Location: None 

User Location: 

Pittsburgh, Pennsylvania 

User Intent: Users are 
looking for pictures of 
kittens. This is a non-local 
intent query. There is no 
obvious user intent to find 
pictures of kittens in 
Pittsburgh. The user 
location plays no role in 
the utility rating. 




See more tlian 5.200,000 images 



The LP is a page full of 
kitten pictures. The result 
block has a nice 
assortment of pictures as 
well. 



OT SH R U V 



Pittsburgh pets classifieds - craigslist 

pittsburgh.craigsiist.org > CL ) Pittsburgh > CDmmunity > pets 
(Pittsburgh..) pic. Sweet senior yeiiow iab BREEZY needs a ioving home! 
FRIENDS Pittsburgh. PA.) pic. Great pets waiting for loving homes! 



{ANIMAL 



This is a local listing of 
pets needing homes in the 
Pittsburgh area. There 
are no pictures of any pets 
directly on this page and 
few pictures on the 
individual listings. 
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Part 3: Block Utility Rating Guideline 



17.0 Overview of Block Utility (BU) Rating 

For every BU rating task, you must: 

• Understand the query (release the task if you can't!). 

• Assign a utility rating to all results. 

• Assign a PQ rating to the landing page of the result block if a PQ slider is present. 

• Identify duplicates (see Section 32.0 ). 

• Flag Foreign Language (F), Porn (P), and Didn't Load (D) results. 

You will develop your own workflow as you master BU rating. You may decide to identify duplicates first or assign PQ 
ratings before utility ratings. 

Remember: If you can't understand the query and user intent, please release the task. For this reason, understanding 
the query is a good first step. 

All blocks in BU rating have a utility slider. Please assign a utility rating for all results. Please assign an OT rating for: 

• Unsolicited Foreign Language (F) results 

• Unsolicited Porn (P) results 

• Didn't Load (D) results 

• Any other kind of useless result 

Some results don't have a PQ slider. If a result block has no PQ rating slider, you do not have to give a PQ rating. 

If there is a PQ slider, please assign a PQ rating based on the landing page. If you are unable to evaluate the PQ of a 
Foreign Language or Didn't Load result, you should assign a PQ rating of lUledium. 



18.0 The Relationship between PQ and Utility 

The utility rating is based on both the query and the result. For this reason, the utility rating slider is labeled "query- 
dependent." You must carefully think about the query and user intent when assigning a utility rating. 

Utility Rating (quer^'-dependent) 




M/A OT 3R R U V 

The PQ rating slider is labeled "query-independent," meaning that it does nof depend on the query. Do not think about 
the query when assigning a PQ rating to the LP. 

Page Quality Rating (query-independent) 
I I ' I ' l' ' I ' I 

hi'A Lowest Medium Highest 
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Here is some guidance about assigning utility and PQ ratings: 

• Useless results should always be rated OT, even if the landing page is high quality. Useless is useless. 

• On-topic, helpful, but low quality results should get lower utility ratings than on-topic, helpful, and high quality 
results. The utility scale encompasses all aspects of "helpfulness," and many users find low quality results 
less helpful than high quality results. Your ratings should reflect this. 

• The U rating should be given to helpful, high quality pages which are a good fit for the query. The U rating 
may also be used for results which are very helpful, medium quality, and have other very desirable 
characteristics, such as very recent information. 

• Do not use the U rating if a page is low quality or has any other undesirable characteristic, such as outdated 
information, or if it is a poor fit for the query. We have very high standards for the U rating. 

• SR is often a good rating for low quality but on-topic pages. However, a page can be so low quality that it is 
useless for any query. Gibberish pages are a good example of this and should be rated OT. 

• The V rating is special and does not depend on the quality of the landing page. If the user intent of the query 
requests a specific page, then that page should get the V rating, even if it is low quality. 



Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 


[how do electric cars 
work], English (US) 

Find information about 
how electric cars work. 


How Does an Electric Car Work? - Yahoo! Voices - voices. yahoo. com 
voices yahoo. com,'havv-cloes-electric-car-work-3761 31 .hfml 

Jun 7. 2007 "Share your voice on Yahoo i websites. Start Here." ... Electric cars look 
very differBnt under the hood than a gasoline-powered car For one thing ... 


PQ: Read this content out 
loud. The article on the LP is 
shallow and unintentionally 
humorous. There are four 
paragraphs of text, but it 
basically says that electric cars 
use electricity and have to be 
charged. 

Utility: This page would be 
helpful for few users. 






LswEsi Medium hligheit 


T 1 1 1" i| 1 

C!T SR R U V 


Mobility scooter - Wikipedia. the free encvclopedia 


PQ: There is a lot of helpful 
MC on the landing page. 

Utility: The query asks how 
electric cars work, but the 
article on the LP is about 
mobility scooters. 


en.wlklpedla.org/wlkl/Mobii ity_s co oter 

it is often referred to as a power-operated vehicie/scooter or electric scooter as weii. ... 
Forward/reverse direction can be controiied by thumb paddies, fnger ... 


Lowest MEdium High Est 




OT SR R U V 
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Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 


[symptoms of 
dehydration], English 
(US) 

Find information about 
the symptoms of 
dehydration. 


Symptoms of Dehydration | Signs of Dehydration 
wsifiV syniptonisofd6hYcJr3tion corti' 

Know the Signs and Symptoms of Dehydration and be prepared to prevent, identify and 
treat dehydration. 


PQ: This is a YMYL topic. The 
page has many characteristics 
of low quality: no contact 
information, no indication of 
who wrote the contact, no 
evidence of medical 
expertise/authority, and heavy 
monetization from Ads which 
distract from the MC. This 
page is not trustworthy. 

Utility: Even though the article 
is topical, the information is not 
reliable and potentially 
misleading or dangerous. 






Lrj^veat Medium Highest 


OT SR R U V 


Dehydration: Symptoms - IV1avoClinic.com 

wmv.mayoclinic.com > ... > Dehydration > Basics 

Symptoms. By Mayo Clinic staff. Mild to moderate dehydration is litely to cause: Dry. 
sticky mouth: Sleepiness or tiredness — children are likely to be less active ... 


PQ: This is a YMYL topic. 
This page is on a highly 
authoritative medical website 
and has a lot of reliable and 
accurate IVIC. This page is 

\/or\/ tri ict\A/rtrth\/ 
vtJi y 11 uoiwui LI ly . 

utility: This result would be 
very helpful for most users. 






LL3-*VES-t MEdium Highest 


OT SR R U V 



19.0 Queries with Multiple Interpretations and Intents 

Some queries really only have one meaning. Consider the query [iphone], English (US). There may be different user 
intents for this query (Do-Know-Go) and there are different models of iPhones, but all users are basically referring to 
the same thing: the phone made by Apple, Inc. 

Some queries truly have different possible meanings. We refer to these different meanings as query interpretations. 

When giving utility ratings for results involving different query interpretations, think about how likely the query 
interpretation is and how helpful the result is. 

For example, a very helpful result for a dominant interpretation should be rated U, because it is very helpful for many 
or most users. Some queries with a dominant interpretation have a V result. 

A very helpful result for a common interpretation may be U or R, depending on how likely the interpretation is. 

A very helpful result for a very minor interpretation may be SR or lower because few users may be interested in that 
interpretation. 

There are some interpretations which are so unlikely that results should be rated OT. We call these "no chance" 
interpretations. 

Reminder: PQ rating is query-independent. Do not think about query interpretation when doing PQ ratings. 
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Query Interpretation 


Example Query and Interpretation 


Highest Possible Utility 
Rating For a Very 


Dominant interpretation: 

Of all the users who type the 
query, most users wouid 
want ttiis interpretation. 


[apple], English (US): Apple electronics. lUlost users who type this 
query want results on Apple electronics. 

[windows], English, (US): The Microsoft operating system. Most 
users who type this query want results on the Microsoft Windows 
operating system. 

[median], English (US): The mathematical formula. Most users who 
type this query want results about the mathematical formula. There 
is no V result possible for this query. 

[guinea pig], English (US): The small furry animal often l<ept as a 
pet. Most users who type this query want results about the animal. 
There is no V result possible for this query. 


VorU 


Common interpretation: 

Of all the users who type the 
query, some users would 
want this interpretation. 


[apple], English (US): The fruit. Some users who type this query 
could want results about the fruit. 

[windows], English (US): The glass paned windows for a home. 
Some users who type this query could want results about glass 
windows for a house. 

[ada], English (US): The American Dental Association, the American 
Diabetes Association, or the American with Disabilities Act. Some 
users could want information about any of these interpretations. 

[mercury], English (US): The car brand, the planet, or the chemical 
element. Some users could want information about any of these 
interpretations. 

[sandals], English (US): The open type of shoe or the chain of 
resorts in the Caribbean Sea. Many or some users could want 
information about either of these interpretations. 


U or R, depending on how 
likely the common 
interpretation is. 


iUllnor Interpretation: 
Of all the users who type the 
query, few users would 
want this interpretation. 


[ada], English (US): The Atlanta Development Authority or the 
American Darters Association. Few users would want information 
about either of these interpretations. 

[mercury], English (US): The Mercury Magazine (published by the 
Astronomical Society of the Pacific) or Mercury Records (a record 
label in the U.K.). Few users would want information about either of 

these interpretations. 

[hot dog], English (US): "Hot Dog," a movie from 1984. Few users 
would want information about this interpretation. 


R or SR 

Results for less likely 
interpretations should get 
lower ratings because 
they are helpful for fewer 
users. 


"No chance" 
Interpretation: An 

interpretation so minor that 
almost no one would ever 
want it. 


[guinea pig], English (US): A pig from New Guinea, which is an 

loidi lu LrUUi III y lULrdidJ i icdi /^uoii ctiid. i i id c yji KJUdUiy di c |jiyo ii i 

New Guinea, but it is extremely unlil<ely that the user typing the 
query would have that interpretation in mind. 


OT 

No users are interested in 
"no chance" 
interpretations. 
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20.0 Specificity of Queries and Landing Pages 

Some queries are very general and some queries are specific. Here are some examples that compare levels of 
specificity of English (US) queries: 



Query 


More Specific Query 


Even More Specific Query 


[chair] 


[dining room chair] 


[ikea "henriksdal" highback upholstered chair] 


[cameras] 


[Nikon cameras] 


[Nikon dSOOO sir] 


[Toyota] 


[Toyota hybrid] 


[Toyota Prius 2010] 


[library] 


[Harvard library] 


[Harvard Anthropology library] 


[interview questions] 


[interview questions for teachers] 


[practice interview questions used for Teach For America] 


[discount stores in houston] 


[walmart stores in houston] 


[walmart 9555 South Post Oak Road houston] 


[restaurants] 


[Chinese restaurants] 


[takeout Chinese restaurants in downtown Austin] 



Results for specific queries are easier to rate on the utility scale because we know more about what the user is looking 
for. Giving a utility rating for results for general queries can be difficult. As always, your rating is based on how helpful 
the result is for the query, not the specificity fit. 

When the query is a broad category, such as [cafes] [restaurants] [hotels] [books] [tourist attractions in paris] etc., 
popular and prominent examples may be considered very helpful. 



Of course, PQ rating is not based on the query, so specificity is not a concern for PQ rating. 



Proprietary and Confidential - Copyright 2014 



133 









Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 








[credit cards], English 
(US) 

Users are probably 
looking to sign up for a 
credit card online, or 
want to research credit 
cards before signing up. 


Credit cards & Rewards credit cards 1 Viss USA 
UBa.visa.cc ■ cards ''credit.' 

Whether you are lookirg for a premium rewards card or a tradition credit card. Visa 
has something to fit your needs. 


Even though there are many options, the 
page for an individual company that 
offers different cards could be very 
helpful. 

PQ: Visa is a well-known credit card 
company with a good reputation. 

Utility: This LP is more specific than the 
query, but it would still be helpful for 
many or most users because Visa is a 
popular credit card company. 






Lcxivest Medium Highest 


OT 3R R U V 


Credit cards and Credit card Offers - Apply Online 

htips' :, c: cvaciii CTiriiF,' 

See all the ways you can ear and enjoy credit card rewards with the Discover it credit 
card, Vne game cl^anging cash back credit card 


Even though there are many options, the 
page for an individual company that 
offers different cards could be very 
helpful. 

PQ: Discover is a well-known company 
that offers services for credit cards, 
banking, and loans, and has a good 
reputation. 






Lmivest ^tedi um Hig hest 


1 ■ 1 ■ 1 ■ 1 ■ 1 ■ t • * ■ 1 

OT SR R LP V 


Utilitv Thi^ 1 P i*? mnrp *?nprifir than thp 
query, but it would still be helpful for 

mQn\/ nr m/ict i icorc hor^Qiico l~^ic^/"i\/or ic 
llially Ul IliUoL UbtJio UcOdUoc l_/lbOUVcI lo 

a popular credit card company. 


Best Credit Cards of 2014: Reviews^ Rewards and Offers 

NerdWallet's credit card experts rank the best credit cards out there. ... liking, take a 
look at this more comprehensive list of cards for consumers wilh no credit. 


This page offers a list of top credit cards 




in a variety of categories. 

PQ: This website helps consumers make 
money decisions. It has a good 






reputation and has been recommended 
by CNNMoney and The New York 
Times. 


Lowest Medium Highest 


t-.-t-.-t-.-!|-r-t-r-).-.-^.^^-|-.r-t 

OT SR R U V 


Utility: This LP fits the query. It would 
be helpful for most users. 


Unian Plus Credit Card for Union Membei^ 

v.v; w, union plus . o rg 'rti o n ey- c red it 'ere d i t-c a rd 

The Union Plus Credit Card is designed for union meinbers. The credit card offers 
financial hardship assistance^ strike and layoff benefits. Apply for the Union ... 


This page describes a credit card that 
requires union membership. 

PQ: This company is an expert on its 
own credit card, which is issued by 
HSBC Bank, a well-known financial 






services organization. 

Utility: Since the credit card requires 
union membership, the page would be 
helpful for some users. 


L{9west Medium High>est 


OT SH R U V 
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Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 


[hotels], English (US) 

Users are probably 
planning a trip, but this 
query is very general 
and vague. 


Hotels. Hotel Rooms, Discount Hotel Rates & Cheap Motels on Or. . 


This is a popular travel aggregator 
website, and the hotel page on the site 
can help users find hotels in the U.S. 
Users can read reviews, compare hotels, 
or make a reservation. 

PQ: Orbitz is a popular website with a 
good reputation. 

Utility: This LP fits the query. It would 
be very helpful for most users. 


WWW orbitz comrhotfilsi' 

Find discount hotels, motels, and cnesp hotel rooms witn hotel reviews on Orbitz: Get 
our lowest hotel rates on our entire selection of hotels or automatic cash ... 




1 1 1 1 t 1 i 1 


r— t f— " 1 

Lowest Medium HighEst 


OT SR R U V 


Marriott Hotel Reservations | Find Your Perfect Hotel Room 


This is a very popular hotel chain with 
hotels available in the majority of the 
U.S. at many different price points. 

Even though the list of possible hotel 
chains is long, the homepage of an 
individual chain that offers different 
prices, features, and location options 
could be very helpful. 

PQ: The IVIarriott website gives 
information on Marriott hotels, a popular 
chain of hotels. Marriott is an expert on 
Marriott hotels and the information is 
highly authoritative. 

Utility: This LP is more specific than the 
query, but it would still be helpful for 
many or most users because the Marriott 
is a popular chain of hotels. 


mw.marrlott-com/ 

Easily bool<your ne.Kt hotel reservation and travel accommodations at the offcial 
Marriott Hotels website. Find the perfect hotel room for your next adventure ... 


L:jWEst MediurTK Highest 


OT SR R U V 


Courtyard Emeryville Hotel - Marriott 

mw-marrioB com > Court^'ard > Emeryville > Hotel 

The Courtyard Emeryville hotel Is located near Berkeley. San Francisco and downtown 

Oakland restaurants, shops and entertainment venues. 


This is the webpage of the Marriott 
Courtyard hotel in Emeryville, California. 

PQ: The Marriott website gives 
information on Marriott hotels, a popular 
chain of hotels. This page is 
professional, well designed, and 
functions well. Marriott has a good 
reputation and is an expert on Marriott 
hotels, making the information on this 
page highly authoritative. 

Utility: The LP is too specific for the 
query, but this is a well-known brand and 
users can navigate to other Marriott 
hotels from this page. Few users would 
find this page helpful. 


1 -■ t ' t ' t ^ 

L-nwest MEdlum High Est 


OT SH R J V 
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Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 




Target.com : Furniture, Babv, Electronics, Tovs 
vfmv.target.com.i' 

Expect more pay less with Target. Spend $5D, get free shipping on over 500K items. 
Choose from a wide seiection of furniture, baby, eiectronics. toys, shoes, c. 


The LP is tlie official Target liomepage. 

PQ: Target is a popuiar store whicti seiis 
many different products. Target lias a 
good reputation, but is not an expert or 
autliority on mucin of what it sells. 










Lowest Medium Highest 


,..,^1.1. If ill i^ii- i| <|- ., |r— ..^ 

or 5R R LT V 


Utility: The landing page fits the query. 




Target : Store Locator : Find Store 

wtw.ta rg et com;'store-i oc atonf n d -stores 

Enter your city and state or ZiP Code to find a Target store. Oniy search for stores ... 
Plan Your Target Trip and Save Groceries, coupons and more. Start Now > ... 


The LP is the "store locator" page on the 
Target website. 

PQ: Target is the expert about where its 
stores are located. This page is the 
most authoritative page on this topic. 














Utility: The LP is more specific than the 




Lijivest Medium Highest 


OT SR R U V 


query, but many or most users would be 
interested in this page. 


[Target], English (US) 
Go to target.com or find 

CL i\J\jCii I Cll OLL/IC. 


Babv Gear: Furniture, Clotliing, Strollers, Feeding : Target 

wm.ta rg et. c om.f c.'baby/-/N-5xtly 

Target Is your online baby central with diapers, clothing, strollers, car seats, cribs, 
furniture, feeding & more. Free shipping on select orders over $50! 


The LP is the "baby" page on the Target 
website. 

PQ: Target has a good reputation, but is 
not an expert or authority on much of 
what it sells. 


Lawest Medium Highest 


OT SR R ir V 


Utility: The LP is more specific than the 
query, but many or most users would be 
interested in this page. 




Electronics: TVs, Audio, iPods, Digital Cameras : Target 

virww.target.eom/c/electronics./-/N-5Alg6 

Shop electronics, televisions, audio, iPods, computers, digital cameras, video games, ceil 
phones & home theater. Free shipping on select orders over S5DI 


The LP is the "electronics" page on the 
Target website. 

PQ: Target has a good reputation, but is 
not an expert or authority on much of 
what it sells. 




Lcrt'est Medium hfighest 


OT SR R LT V 


Utility: The LP is more specific than the 
query, but many or most users would be 
interested in this page. 




Mossimo Supply Co. Juniors Coiored Sl^innv Denim ... : Target 

mw.target.com/p/mossimo-suppiy-co-juniors... 

Find product Information, ratings and reviews for a Mossimo Suppiy Co. Juniors Coiored 
Skinny Denim - Assorted Coloreds. This Mossimo Supply Co. Juniors ... 


The LP on the Target website is for a 
specific type of junior colored jeans. 

PQ: Target has a good reputation, but is 
not an expert or authority on much of 
what it sells. 




Low^ Medium Highest 


OT SR R U V 


Utility: The LP is much more specific 
than the query and would be helpful for 
very few users. 
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Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 


[Honda Accord], English 
(US) 

Users probably want to 
buy a car and are 
interested in finding 
information about tlie 
Honda Accord. Tliere 
are currently two 
models of the Accord 
today (the Accord 
Sedan and the Accord 
Coupe), and a Plug-In 
model will be coming 
out in 2014. 


Choose the 2013 Accord Sedan or 2013 Accord Coupe ,,. - Honda 

s utomobil &&.h 0 n d a . c Qm/a ccord/ 

Compare the performance and features of the stylish Accord Sedan and sporty Accord 
Coupe The Accord combines Honda's standards of quality reliability ... 


The LP is the official Honda Accord 
page. 

PQ: The Honda website gives 
information on Honda cars, a very 
popular car maker. This page is 
professional, well designed, and 
functions well. Honda has a good 
reputation and is an expert on Honda 
cars, mailing the information on this page 
highly authoritative. 

Utility: The LP fits the query. The query 
asl<s about the Accord and the landing 
page is about the Accord. 






Lowest Mediiim Highest 


or SR R U V 


Honda Cars - New and Certified Used Cars from American Honda 


The LP is the official Honda Automobiles 
webpage. There is a prominent "Accord" 
linl< on the page, as well as a lot of 
helpful features for users interested in 
Accords. 

PQ: Honda is an expert on Honda cars, 
making the information on this page 
highly authoritative. 

Utility: The LP is more general than the 
query, but the information on the page 
would be very helpful for many or most 
users. 


automobiles honda com ' 

Honda Cars - Home of Information on the latest Honda cars and SUV models. Certif ed 
Used Cars. Photos, videos, and technical specs for all car models from ... 






LtjwEst Medium Highest 


OT SF? R U V 


2013 Honda Accord Coupe Overview - Official Honda Site 


The LP is the official page of the Accord 
Coupe. 

PQ: Honda is an expert on Honda cars, 
making the information on this page 
highly authoritative. 

Utility: The LP is more specific than the 
query, but since there are three Accord 
models, official pages for any of the three 
models are U. 


automobiles honda com/accord -coupe.' 

See reviews, features and specs of the 201 3 Honda Accord Coupe at the official Honda 
Website. View pictures and learn about the iatest changes for 201 3. 






LcwEt Medium Highest 


OT S? R U V 


2013 Honda Accord Sedan - Warranty - Official Honda Site 


The LP is the official "Accord Sedan 
Warranty" page. 

a 

PQ: Honda is an expert on Honda cars, 
making the information on this page 
highly authoritative. 

Utility: The LP is much more specific 
than the query. It would be helpful for 
few users. 


a utom obi 1 es h 0 n d a c 0 m/a ceo rd -sed a n./wa rra nty. a spx 

Get warranty information on the 201 3 Honda Accord Sedan at the Official Hone 

Website. 






L-nwEst Mediumi Highest 


OT SR R U V 
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Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 


[Honda Accord], English 
(US) 

Users probably want to 
buy a car and are 
interested in finding 
information about tlie 
Honda Accord. Tliere 
are currently two 
models of the Accord 
today (the Accord 

Qarlan anri thp Arrrirrl 

Coupe), and a Plug-In 
model will be coming 
out in 2014. 


Honda Power Equipment: Honda Generators. Lawn Mowers ... 

pow&rec| uprnent.honda.corn.i' 

Welcome to the official site of Honda Power Equipment. Honda provides you with 
powerful and durable generators, lawn mowers, tillers, trimmers, snowblowers, ... 


The LP is the official Power Equipment 
page on the Honda website. It has 
nothing to do with the Honda Accord. 

PQ: Honda is an expert on Honda power 
equipment, making the information on 
this page highly authoritative. 

Utility: The LP is does not fit the query. 

K/lnQt 1 iQfirQ \A/ni ilH nnt finrl thlQ rPQi lit 

IVIUoL UodO vVVJUlU IIL/L IIIIU Llllo ICoUIL 

helpful. 






LswE&t MEdium Highest 


OT SR R U V 


[chicken recipes], 
English (US) 

Users probably want to 
make a chicken dish 
and are looking for 
some recipes to choose 
from. Users probably 
expect and want a list of 
recipes. 


Chicken Recipes: Chicken Breasts, Fried Chicken ... - Food Net... 


This LP has many chicken recipes (with 
reviews) on a popular recipe website. 

PQ: Food Network is a popular website 
with a good reputation for providing high 
quality recipes and cooking information. 

Utility: This LP fits the query. It would 
be very helpful for most users. 


mw,foodnetwork.com..'topics,.''(;hlcken,fincie.>:-html 

Get help with your chicken dinner from FoodNetwork.com. Find easy chicken meals, 
including recipes for fried chicken chicken salad chicken soup and more 


1 ' 1 ■ 1 ' t ^ 

Lowest MEdium Highest 




OT SR R U V 


Chicken Parmesan Recipe : Tvler Florence : Recipes : Food Network 


This LP has a single recipe for chicken 
parmesan. 

PQ: Food Network is a popular website 
with a good reputation for providing high 
quality recipes and cooking information. 

Utiiitv Thi*? i^ 3 nnniil^r tvnp nf rhirkpn 
recipe on a popular website, but the LP 
is more specific than the query. Some or 
few users would find this page helpful. 


WAf.foodneti-vorK.CDm > Recipes > ChicKen 

Get this all-star, easy-to-follow Food Network Chicken Parmesan recipe from Tyler 
Florence. 


1 - 1 1' 




La-iVES-t M Ed 1 u mi Highest 


OT 3R R U V 


Best Fried Chiclten Recipes - Allrecipes.com 

allreclpes.com > Recipes > Meat and Poultry > Chicken 

Top 20 Fried Cliicken Recipes from Allrecipes.com as rated and reviewed by ... These 
buttermilk fried chicken legs are eyXra crispy because you double dip them ... 


This LP has 20 recipes for fried chicken, 
a popular chicken dish. 

PQ: Allrecipes.com is a popular website 
with a good reputation for providing high 
quality recipes with lots of reviews. 

Utility: Even though there are 20 
different recipes on the LP, they are all 
for the same basic dish. Therefore, this 
LP is also more specific than the query. 
Some or few users would find this page 
helpful. 






Lwv^ hAedijm KighE^t 


OT SR R U V 
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Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 








[chicken recipes], 
English (US) 


ChiickenRecipes.com: The Leading Chicken Recipe Site on the Net 
chickenrecipes.com/ ^ 


This is a fake search page for chicken 
recipes. 


Users probably want to 
make a chicken dish 
and are looking for 
some recipes to choose 

frnm 1 l^^prt? nrnhphlv 

IIVJIII. t^iocio ^l\JK>CXK>ty 

expect and want a list of 


chickenrecipes.com has been connecting our visitors with providers of Beer Brevi^ing, 
Beer Making. Chef Schools and many other related ser^/ices for nearly 10 ... 


PQ: Ads should never disguise itself as 
the MC of the page. Pages with Ads that 
are designed to look like MC should be 
considered deceptive, making this fake 






cpqrph nqnp L OW^^t mialitV 


recipes. 


L[r'.vest lAsd \um Hig hest 


OT 3R R LF V 


Utility: It should be rated OT as it would 
be helpful for no users. 


[digital cameras], 
English (US) 

Find information about 
digital camera brands 
and options before 
buying a camera. 


Diaital cameras: compare diaital camera reviews - CNET Reviews 


IVIany users research purchases 
carefully. This LP is a cnet.com "Digital 
cameras" review page, with information 
about many different digital cameras 
organized by price, manufacturer, and 
camera features. 

PQ: Cnet.com is a popular website 
known for providing helpful reviews of 
consumer electronics products. This 
article has a lot of very helpful IVIC and 
SC. 


revie',vs cnetcom/digital-cameras/ 

Digital camera reviews and ratings, video reviews, user opinions, most popuiar digital 
cameras, camera buying guides, prices, and compansons. 


LoiVES't ME-dium HighEit 




t ' 1 ■ 1 • 1 ' 1 ■ 1 • * • 1 ■ 1 
OT 3R R U V 


Utility: This LP fits the query. The query 
asks for digital cameras and the LP is 
about digital cameras. 


Nikon Coolpix P520 - Digital cameras - CNET Reviews 


This LP is a cnet.com review page about 
the Nikon Coolpix P520. 

PQ: Cnet.com is a popular website 
known for providing helpful reviews of 
consumer electronics products. 

Utility: It is a comprehensive review of a 
popular type of camera made by a 
respected brand, but it is more specific 
than the query. 


reviews-cnet.com > Reviews > Digital cameras 

Jan 28, 2013 - Latest models bulk up on Wi-Fi. megapLxels. and zoom. 






Low^ Medium HigiiEst 


or SR R u 



21.0 Utility and Freshness 

Some queries demand very recent or "fresh" information. Users may be looking for "breaking news," such as an 
important event or natural disaster happening right now. Imagine a user in Kansas typing the query [tornado], English 
(US) because a big storm is coming. Information about last year's tornado season is not helpful. Users need 
information about tornadoes headed their way right now! 

Queries for recurring events, such as elections, sports events, TV shows, conferences, etc., also demand current 
information. Assume users are looking for the most recent or current information about the event, unless the query 
indicates otherwise. For example, if the Olympics are happening right now, users typing the query [Olympics] want 
information about the current Olympics, not results from years ago. If the next Olympics are a few months away, users 
are probably interested in the next upcoming Olympic Games. 

Some queries demand current information, such as [population of paris], [next federal holiday], etc. Requests for 
prices such as [airfare from ny to sfo], should be interpreted as asking for a current price. Unless otherwise stated, 
users usually want information about the most recent model/version for product queries, such as [iphone], [windows 
operating system], [toyota camry], etc. 
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Here are some different types of queries demanding current/recent results: 

• "Breaking news" queries sucin as [tornado] 

• Recurring event queries sucin as [Olympics], [american idol], [redsox schedule], [tax forms], [elections] 

• Current information queries such as [population of paris], [amount of u.s. debt], [airfare from ny to sfo], 

• Product queries such as [iphone], [toyota camry], [windows operating system] 

When a query demands current or recent content, only pages with current, recent, or updated content should get high 
utility ratings. Remember that we care about the content, and not the date the page was created. For these queries, 
pages about past events, old product models, old prices, outdated information, etc. are not helpful and should be 
considered "stale." When the query demands recent information, stale results should be given low utility ratings 
because they are not helpful for users. In some cases, stale results are useless and should be rated OT. 



Freshness is generally less of a concern for PQ rating. "Stale" pages can have high PQ ratings. For example, some 
highly reputable news websites maintain "archival" content. However, unmaintained/abandoned "old" websites or 
unmaintained and inaccurate/misleading content is a reason for a low quality rating. 



Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 


[nasa mars landing], 
English (US) 

On the date this example 
was written, NASA had 
just landed the 
"Curiosity" rover on 
Mars. Most users are 
probably looking for 
breaking news and 
pictures about the event. 


Curiositv Rover Lands Safelv on Mars - MYTimes.com 

v™w.nytimesxom/.../curiosity-rover-lancls-safely-on-mars.ht... 

Aug 6. 2D12 - One of thefrsttest images from NASA's Mars Curiosity rover that Ineiped 
... Engineers chose not to use the tried-and-true landing systems. ... 


1 1 iio i-r 1 icio cti 1 cti liLiic; ctUUUL 11 ic 

"Curiosity" rover landing, 
published on the same day as 
the landing. 






Lowest Medium HighKt 


OT 3R R U V 


NASA's Curiosity Rover Sets Off for Mars Mission - NYHmes.c. .. 


This LP has an articie about the 
iaunching of the "Curiosity" rover, 
published nine months before its 
ianding on Mars. The content on 
this page would be considered 
stale for the query. 


mw.nytimes.com/.. /nasas-curiosity-rover-sets-off-for-mars -mission 

Nov 2B. 201 1 - CAPE CANAVERAL, Fia. (AP) — Theworid's biggest e.>ctraterrestriai 
expiorer, NASA's Curiosity rover, rocKeted toward IVlars on Saturday on a ... 






LnwEst Medium Highest 


OT 3R R U V 


[los angeles traffic], 
English (US) 

Users are looking for 
current information 
about traffic conditions in 
Los Angeles. Even 
pages about traffic 
conditions the day before 
would be considered 
stale for the query. 


Your commute: Live L A. traffic conditions - iatimes.com 


This LP shows traffic conditions 
in Los Angeles on August 7, 
2012, which is the day this 
example was written. 


latimesblogs.latimes,com.''....i''your-commute-live-la-traffic-conditions-. . 

Your commute: Live L.A. traffic conditions. August 7, 2012 1 6:30 am. increase te.>S 

size decrease text size. Los Angeies. Long Beach . East L.A./Pasadena ... 




1 i 1 1 ■ t ' t - ^ ■ i 1 

OT 3R R U V 


Lowest l^iledium HighE^t 


Your commute: Live LA traffic conditions - Biogs - Los Angeies Ti... 


This LP shows traffic conditions 
in Los Angeles on August 6, 
2012. it is stale and useless for 
the query even though it is only 
one day oid. 


iatimesbiogs.latimes.comi'.../your-commute-iive-ia-traffic-conditions- 

Your commute: Live L.A. traffic conditions August 6. 2012 | 6:30 am. increase teA 

size decrease tex1 size. Los Angeies Long Beach East L.A./Pasadena ... 






L-DWEst Medium Highest 


OT SR R U V 
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Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 


[nobel peace prize 
winner], Englisli (US) 

Find tlie name of the 
most recent winner of 
tliis prize. Tills is a 
Know query. 
At the time this example 
was written, 201 1 was 
the most recent year that 
the Noble Peace Prize 
was awarded. 


Nobel Peace Prize Awarded to Three Activist Women 

mw. nytimes-CDnii' 'nobel-peace-priie-johnson-sirlear-gbDW... 

Oct 7. 2D1 1 - LONDON — The Nobel Peace PriEe for 201 1 was awarded on Friday ... 

The winners were President Eiien Johnson Sirieaf of Liberia — the frst ... 


This LP on the New York Times 
website has information about 

thp mn'^t rprpnt winnpr nf thi^ 

LIIC IIIWOL I^Odll VVIIIIId \Jl LIIIO 


LffiVEst ^4ediurT» Highest 


OT SR R U V 


prize. 


BBC NEWS 1 Europe j Nobel for anti-poverty pioneers 

news.l3l)C-Co.ult'2/hi/europe/6047020-Stm 

... Muhammad Yunus and the Grameen Bant; are awarded the 2(106 Nobel Peace Priie 
... Last Updated: Fnday. 13 October 2006. 16:13 GMT 17:13 UK... The BBC's Lars 
Bevanger in Osio says this year's winner caught most there by surprise. 


This LP on the BBC website has 
Information about the 2006 
winner of this prize. This page Is 


L^jWESt Medium Hishest 


OT SR R U V 


stale for the query. 


[60 minutes], English 
(US) 


60 Minutes Interviews with Obama and Romnev, September 2012 ... 

www.cfr.org/us-eiection-2D1 2/S0-minutes...j'p2ai 07 

Sep 23, 2012 - CBS' 60 Minutes interviewed President Obama and Repubiican 
presidentiai candidate Mitt Romrey for a program ttiat aired on Septemtier 23, ... 


This LP Is an article about the 
most recent episode of 60 
Minutes, which aired on 


60 IVIinutes is a popular 
weekly TV show. Users 
probably want to watch 
or find information about 


Lrjwest Mediumi Highest 


OT SR H U U 


September 23, 2012 








the most recent 

episode. At the time this 
example was written, the 
show that aired on 
September 23, 2012 was 
the most recent episode. 


"60 IVIinutes" Tiie Spvmaster Spealcs/The Piaver/The Year of ... - i... 

mw.imdb.com/titieytt2408648..' 

60 Minutes: Season 44, Episode 52. The Spymaster Speal<s/The Piayer.'The Year of 
Adele (16 6ep. 2012). TV Episode - 60 min - Documentary | News ... 


This LP Is about an episode of 
60 Minutes that aired on 
SeDtember 16 2012 Users 
might be looking for the episode 
from the previous week. 


Lo^vest Mediumi HighKt 


OT SR R LF V 




Gymnastics at the 2012 Summer Olympics - Wikipedia. the free ... 

en.wikipedia.org/wil<i/Gyninastics_at_the_2012_Summer_Olympics 

Main articie: Gymnastics at the 2012 Summer Olympics - Quaiircation. Quaiifcation 

was based on the results of the 201 1 Worid Rhythmic Gymnastics ... 


This LP has the 2012 Olympics 
gymnastics results, which was 
the most recent summer 
Olympics at the time this example 
was written. 


[Olympics gymnastics 
results], English (US) 


1 1 ^ 1 ' 1 

L::\VESt Medium HighEst 


1 1 1 1 1 1 ¥ 1 1 

OT 3R R U V 


Users probably want to 
find a page that displays 
gymnastics results from 
the most recent 
Olympics. 








Gymnastics at the 2008 Summer Olympics - Wikipedia, the free ... 

en .wi l<iped i a org/ /Gym n a sti cs_at_th e_2 0 0 8_S u m m er_0 iym . . . 

At the 2008 Summer Oiympics. three different gymnastics discipiines were contested: 

artistic gymnastics, rhythmic gymnastics and trampaiine. The artistic ... 


This LP has the 2008 Olympics 
gymnastics results. It is stale for 








u It; LjUc;! y . 




Lowest MEdium Highest 


1 1 1 ^i._.._it_._,|_.^f_.^4-i f 
OT SR R U V 
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Sometimes, queries don't demand tine most recent information. For example, some queries, sucin as [birtlndate of 
ronald reagan] and [atomic weigint of gold] request basic facts that are well understood, established, and unlikely to 
change. 

Finally, for some queries, there may be a "newsy" or recent information user intent as well as a more "timeless" 
information user intent. Users issuing queries for celebrities or politicians may be interested in biographical information 
or may be looking for the latest news or gossip. 

Note: Information about almost any subject can become stale or outdated over time. Please use your 
judgment. Clearly outdated or stale pages should receive low utility ratings, unless the query is "asking" for the 
outdated content. 



Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 


[arthritis], English (US) 


Arthritis: Medline Plus 

mw.nlm.nih.gov/medlineplus/arthritis.html 

if you feei pain and stiffness in your body or have trouble moving around, you might have 
arthritis. Most l^inds of arthritis cause pain and sweliing in your joints 


Medline Plus is an authoritative 
website that regularly updates its 
content. Even though there is no 
date on the page indicating when 
it was updated, we can assume 
the content is up-to-date. 


Users are looking for 
information about this 
disease. Because there 
is ongoing research done 
on treatments and cures 
for most diseases, users 
are probably looking for 
authoritative, up-to-date 
information on this 


Lowest Medium KighEst 


OT SR R U V 


Arthritis Drug Approved for Expanded Use - New York Times 

wwm. nyti m es. c om/2 0 0 S/0 5/2 0/bu si n ess/2 0 d ru g . htm 1 

May 20. 2006 - Federal regulators approved wider use of Johnson & Johnson's drug 
Remicade but also said that a new warning was being added. 


This 2006 article would be 
considered stale for this medical 


subject. 


Lowest Medium Highest 


OT SR R u V 


query. 


[when was franklin 
delano roosevelt born], 
English (US) 


Franklin D. Roosevelt Bioaraphy - Facts, Bitthdav, Life Storv ... 

biography.comi'peoplei'Tranl<lin-d-roosevelt-g463381 
Biography. com highlights President Franklin D. Rooseveit. Find out more about ... 
The Biography Channel website; 2013 [cited 201 3 Feb 14] Available from; ... Franl^lin 
Delano Roosevelt vras born on January 30. 1 8S2. Into a wealthy family. 


This LP has the information 
users are looking for. Notice 
that the requested information is 






even displayed in the snippet. 


Users want to find this 
well understood fact. 


LffsVEst Medium htigiiest 


OT SR R U V 






We couldn't find a stale page for this query. 




[barack obama], English 
(US) 


President Obama Inauguration - NYTimes.com 

mw.nytimes.com.i'-..i'presldent-otiama-inauguration.html?...all 

Jan 20. 201 3 - The quiet ceremony before Monday's public pomp, forced by a quirk of 

the constitutional calendar, started President Obama's second term with ... 


This article contains up-to-date 
news about Obama at the time 
this example was written. 






Barack Obama is the 
President of the United 
States. We will assume 
users are looking for 
news, current or 
biographical information 
about him, images, etc. 


LuwEst Medium Higiiest 


OT SR R U V 




Obama Chooses Biden as Running Mate - NYTimes.com 

www.nyll mes. c om/20 0 8/0 6.e4/u s/pol iti c s/24 bi d en . html 

Aug 23. 2008 - WASHINGTON — Senator Barack Obama has chosen Senator Joseph 
R. Biden Jr. or Delaware to be his running mate, turning to a leading ... 


This LP has an article from 2008 
about Barack Obama choosing 








Joe Biden as his running mate. 




Lowest Mediumi Higtiest 


OT SR R U V 





Note: The date the page was created may be different from when the content was last updated or modified. When 
content is updated, the page will sometimes show the date of the update, not the date the page was created. 
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If you are curious about the content of a page, see here to try the "Wayback Machine" on the Internet Archive. Not all 
pages are available, but this tool may help you understand how some pages were created and how their content has 
changed over time. 



22.0 Misspelled and Mistyped Queries and Results 



22.1 Misspelled and Mistyped Queries 



You will notice that some queries are misspelled or mistyped. Here are some examples of queries that are obviously 
misspelled: 



Misspelled Query 


Query Interpretation 


[federal expres], English (US) 


The only reasonable query interpretation is the company named Federal Express. 


[my sapce], English (US) 


The only reasonable query interpretation is the website MySpace. 


[the ecomonist], English (US) 


The only reasonable query interpretation is the news and economics publication. 


[expdeia], English (US) 


The only reasonable query interpretation is the travel website. 


[new england patroits], English US) 


The only reasonable interpretation is the NFL football team. 


[byonce knowles], English (US) 


The only reasonable interpretation is the famous singer/actress, Beyonce Knowles. 



Some misspelled or mistyped queries are more difficult to interpret. Use your judgment and do query research. 



For obviously misspelled or mistyped queries, you should base your rating on user intent, not necessarily on exactly 
how the query has been spelled or typed by the user. 

For queries that are not obviously misspelled or mistyped, you should respect the query as written, and assume 
users are looking for results for the query as it is spelled. 



22.2 Name Queries 

Consider the query [John Stuart], English (US). There is a very famous Jon Stewart, the comedian and host of a 
popular U.S. television show. However, we should not assume that the query [John Stuart] has been misspelled. 
There are many people named John Stuart. We will respect the query as written and assume the user is looking for 
someone named "John Stuart." 
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Query and Query 
Interpretations 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 




Jamie rox - Jazz ouiiarisi ano uornpossr 
jamiefoxguitar.com/ 

... Press ■ News ■ Calendar ■ Photos ■ Buy ■ Guestbook ■ Links. Join the emaii listl Jamie 
Fox Jamie Fox Jamie Fox: Contact. 917 670-2594. ©2013 Jamie Fox. 


Official homepage of 
Jamie Fox, the guitarist. 




L-Gwest Medium Highest 


OT a) H J V 




Oamie fox], English (US) 

There are many 
reasonable 
interpretations for this 
query including the 
guitarist named Jamie 
Fox, Jamie Fox the 
politician, Jamie Fox 
Photography, Jamie Fox 
the actress, and regular 
people named Jamie 
Fox. 

Helpful results for more 
prominent people named 
Jamie Fox should 
receive higher ratings 
than helpful results for 


Jamie Fox Photography - St. Louis, MO 
Jamiefoxphotograpliyxom/inclex2 php 

A St. Louis chiidren's photographer speciaiiiing in famiiies and youngsters of aii ages. 


Official homepage of 
Jamie Fox 
Photography. 


1 ' 1 ' 1 ' ^ 1 

LswE&t Medium Highest 


OT BR H U V 


Jamie Fox - Wikipedia, the free encvclopedia 

en.wikipeclia.org.i'\viki/Jamie_Fox 

James P. "Jamie" Fox (born 1954) is a New Jersey poiitician and poiitical strategisL ... 
He grew up in Union Township. Union County. New Jersey and attended ... 


Wikipedia page on 
Jamie Fox, the New 


L<yrtest Medium Highest 


OT Hi H J V 


Jersey politician. 


less prominent people 

namprl .lamip Foy 

1 lul 1 VJCll 1 lie 1 wA. 

Because Jamie Foxx is 
such a famous actor, we 
will consider Jamie Foxx 


Jamie Fox - IMDb 

mnv.imdb.com.('name/nm 1 785209/ 

indie i-lorror i-lits VoiumeS (video) (as Jamie IW. Fox). 2009 Page Turner (short). 2008 
Virgin Wanted (short) Amy. 2007 The IVIemory Thief. 2007/i Out of Focus ... 


IMDb page about Jamie 
Fox, the actress. 


to be a minor 
interpretation, not off- 
topic. 


Low^ Medium Highest 


i|-n , t , , i 

OT SH R U V 




Jamie Foxx | The Official Website 
vwiv J a m i ef 0 XX. c om/ 

in addition to oversight by Ron Howard, you'll fnd the iikes of Jamie Foxx and Eva 
Longoria getting in on the action. The other directors are Twitter co-founder ... 


Official homepage of 
Jamie Foxx, the actor. 










Ln^vest Medium Highest 


OT BR R U V 
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Query and Query 
Interpretations 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 


[Micheal Jordan], 
English (US) 

There are many people 
named "Micheal Jordan." 

Helpful results for more 
prominent people named 
IVIicheal Jordan should 
receive higher ratings 
than helpful results for 
less prominent people 
named Micheal Jordan. 

Because Michael Jordan 
is such a famous 
athlete/celebrity, we will 
consider Michael Jordan 
to be a minor 
interpretation. 


Micheal Jordan profiles | LInkedIn 

www. 1 i n ked i n . c 0 m/p u b/d i r/M i ch e a l/J ord a n 

View the proflles of professionals named Micheal Jordan on Linkedln. There are 1 6 
professionals named Micheal Jordan, who use Lintedin to exchange ... 


Linkedln page for the 
name "Micheal Jordan." 
The page lists all 
members of the 
Linkedln site named 
"Micheal Jordan." 
Linkedln is a very 
popular social 
networking website for 
professionals in the 
U.S. This result would 
be very helpful for many 
or most users trying to 
find information about 
someone named 
"Micheal Jordan." 


Lim^ Medium Highest 




or SR R U V 


Micheal Jordan | Linkedln 

wurw. 1 i n l<ecl i n . c om/pu b/m i ch e a 1 -J 0 rd a nM 5/ba9/5 43 
Atlanta. Georgia - Facilities Services Professional 

View Micheal Jordan's professional proflle on Linkedln Linkedln is the world's largest 
business nebvork. tnelping professionals like Micheal Jordan discover ... 


Linkedln page for 
Micheal Jordan, a 
farilitip'^ '^prvirp^ 
professional in Atlanta, 
Georgia. 






Lowest Medium Highest 


OT BR H U V 


NBA, com: MichaelJordan Bio 

■mf'iV- nba. com/play enfile/mlchaeljordan. html 

IVIichael Jordan | 23. Season statistics & Notes ■ Season splits ■ Game-by-game stats ■ 
Bio ■ Printable player file. 2002-03. Statistics. PPG, 20.0. RPG, 6.10 ... 


Michael Jordan's player 
profile page on the NBA 
basketball website. 






f— ^ • 1 

Low^ Medium Highest 


OT SR R U V 



22.3 Spelling Suggestion Result Blocks 

When a user misspells or mistypes a query, search engines may display spelling suggestions. 

Users rely on "Did you mean" or "Do you mean" suggestions to tell them when they might be spelling something 
incorrectly and as a way to get better search result pages. "Did you mean" type result blocks are NTRBs, but they do 
have a prominent link to a page of search results for the suggested spelling. 

Your utility rating should reflect both the helpfulness of the suggestion itself and the helpfulness of the LP of the 
suggestion. For every spelling suggestion, please be sure to look at the suggestion and click through to the landing 
page of the suggestion before rating. Sometimes, you will find it helpful to compare the landing page of the suggestion 
to the landing page of the original query. 
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Here is some specific guidance on assigning utility ratings to spelling suggestion result blocks: 



Rating 


Use this rating when: 


Vital (V) 


Do not use the V rating for spelling suggestion result blocks. 


Useful (U) 


Tlie result is very helpful for many or most users because: 

• The query is clearly and severely misspelled. 

• The spelling suggestion is very likely or exactly what users are looking for. 

• The landing page of the suggestion is very helpful. 


Relevant (R) 


The result is very helpful for some users or helpful for many users because: 

• The query is likely or very likely misspelled. 

• The spelling suggestion is the correct spelling for a likely or somewhat likely intent of the 
original query. 

• The landing page of the suggestion is helpful. 


Slightly Relevant (SR) 


The result is helpful for few users. One or more of the following apply: 

• It is likely that the query is not misspelled. 

• The spelling suggestion represents an unlikely interpretation. 

• The spelling suggestion is trivial, such as adding or deleting a space which makes little 
difference to the query and/or the results on the landing page of the suggestion. 

• The landing page of the suggestion is less helpful. 


Off-Topic/Useless (OT) 


The result is unhelpful or useless. One or more of the following apply: 

• The query is clearly not misspelled. 

• The spelling suggestion itself is misspelled. 

• The spelling suggestion changes the meaning of the query. 

• The spelling suggestion is clearly not what the user is looking for. 

• The spelling suggestion is offensive (e.g., uninvited porn, racial slurs). 

• The landing page of the suggestion is unhelpful (e.g., the results are useless for the 
original query). 



Query and Query 
Interpretations 


SERF for the Query, SERP for the Spelling Suggestion, 
Utility Rating 


Explanation 


[irratated], English (US) 

This query has one 
reasonable interpretation: 
the word "irritated." 


SERP for the original query: LP 
SERP for the spelling suggestion: LP 

Did vou mean: irritated 

OT SR R J V 


The query is clearly 
misspelled. The suggestion 
is the correct spelling of the 
query, and the LP of the 
suggestion is very helpful. 


[culcable], English (US) 

The query is misspelled or 
mistyped. The most likely 
interpretation is the word 
"culpable," though perhaps 
this could be a misspelling 
of a type of cable. 


SERP for the original query: LP 
SERP for the spelling suggestion: LP 

Did vou mean: culpable 

OT SR R U V 


The query is misspelled. The 
suggestion is the correct 
spelling of a likely or very 
likely interpretation, and the 
LP of the suggestion is very 
helpful. 
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Query and Query 
Interpretations 


SERF for the Query, SERF for the Spelling Suggestion, 
Utility Rating 


Explanation 


["jack angle" scrap 
metal] , English (US) 

This query has one 

i ntomrotQ tir\ n ■ Iq^^I^ Fnnio 
II ILc;i pi ULdLIUI 1 . \JaLit\ diyic 

& Co., a scrap metal 
company in Los Angeles. 


SERP for the original query: 1^ 
SERP for the spelling suggestion: 1^ 

Did you mean: "jack engle" scrap metal 

or SR R J V 


The query is clearly 
misspelled. The suggestion 
is the correct spelling of the 

qUcly, dllU lilt; l_r Ul lllfc: 

suggestion is very helpful. 


[imetrex], English (US) 

This query has one 
interpretation: the drug for 
treating migraine 
headaches. 


SERP for the original query: 1^ 
SERP for the spelling suggestion: 1^ 

Did you mean: imitrex 

OT SR R J V 


The query is clearly 
misspelled. The suggestion 
is the correct spelling of the 

qUciy, allU Lliy l_r Ul lllc 

suggestion is very helpful. 


[utube], English (US) 
This query has one 

lUdbUi IdUlc iiiiyrpryLdiiuii. 

the youtube.com website. 


SERP for the original query: 1^ 
SERP for the spelling suggestion: 1^ 

Did vou mean: youtube 

OT SR R J V 


The query is clearly 
misspelled. The suggestion 
is the correct spelling of the 

qUciy, dllU Lllc Lr Ul lilt? 

suggestion is very helpful. 


[vanilla cupcake mix make 
1 cup cake], English (US) 

The meaning of the query 
is clear. 

There is a misspelling or 
typo: cupcake is the 
correct spelling. 


SERP for the original query: LP 
SERP for the spelling suggestion: 1^ 

Did you mean: vanilla cupcake mix make 1 cupcake 

OT SR R U V 


The query has a mild 
misspelling and the spelling 
suggestion is technically 
correct. 

The results on the LP of the 
suggestion are similar to the 
results for the original query. 


[one persons wealth 
raises the value of money] 

The meaning of the query 
is clear. 

There is a misspelling or 
typo: "persons" should be 
written "person's." 


SERP for the original query: 1^ 
SERP for the spelling suggestion: 1^ 

Did vou mean: one person's wealth raises the value of monev 

OT SR R U V 


The query has a misspelling 
and the spell suggestion is 
technically correct. 

The results on the LP of the 
suggestion are similar to the 
results for the original query. 
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Query and Query 
Interpretations 


SERF for the Query, SERF for the Spelling Suggestion, 
Utility Rating 


Explanation 


[twilight new moon sound 
track] 

The query intent is clear. 

There is a possible mild 
misspelling: "soundtrack" 
is the more common and 
possibly preferred 
spelling. 


SERP for the original query: 1^ 
SERP for the spelling suggestion: 1^ 

Did vou mean: twilight new moon soundtrack 

OT SR R U 


"Soundtrack" may be the 
more common and preferred 
spelling, but "sound track" is 
not technically incorrect. 

The LP of the suggestion is 
perhaps a bit more helpful 
than the LP for the original 
query. 

This suggestion is helpful for 
some or few users. 


[cynderella], English (US) 

There are several 
interpretations for the 
query as spelled, including 
a bridal shop, an 
entertainment agency, a 
cake company, etc. 


SERP for the original query: l_P 
SERP for the spelling suggestion: LP 

Did VOU mean: cindereUa 

> 1 T 1 1 1 1 1 1 

OT SR R U V 


It is likely that the query is not 
misspelled. The results on 
the landing page of the 
suggestion probably do not 
match the intent of the 
original query. 

However, this suggestion 
could be helpful for a few 
users. 


[he likes me a lot but 
doesn't want to get hurt], 
English (US) 

This query is not 
misspelled. 


SERP for the original query: LP 
SERP for the spelling suggestion: l_P 

Did vou mean: he likes me alot but doesn't want to get hurt 


The suggestion is misspelled 
even though the original 
query is not. 


OT SR R U V 


[girl in pink dress plays 
outside], English (US) 

This query is not 
misspelled. 


SERP for the original query: LP 
SERP for the spelling suggestion: LP 

jjiu you mean. 9111 iri pinK aress piayso ouisiae 


The original query is clearly 
spelled correctly. This 
suggestion is useless. 


OT SR R U V 


[how shades are cut], 
English (US) 

This query is not 
misspelled. 


SERP for the original query: 1^ 

R P fnr thp Qnplli nn qi innPQtinn ■ 1 P 

0 1 — III l\Jl LI IC 0|JC?lllll^ oU^^CoLIVJI 1 . 1— 1 

Did you mean: how shadows are cut 


The original query is clearly 
spelled correctly. The 
suggestion changes the 
meaning of the query. 


OT SR R Lf V 


[people can take a wall 
and turn it into a tv], 
English (US) 

This query is not 
misspelled. 


SERP for the original query: LP 
SERP for the spelling suggestion: J_P 

Did you mean : people can take a walk and turn it into a tv 


The original query is clearly 
spelled correctly. The 
suggestion changes the 
meaning of the query. 


OT SR R J V 
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23.0 Non-Vital Results for URL Queries 



Raters sometimes ask the question, "For a well-formed working URL query, are the only acceptable utility ratings for a 
result either V or OT?" The answer is no. There can be other helpful results for URL queries. 

Some users issue URL queries to find information about a website, such as reviews or recent news. We 
recommended this to you as one method of reputation research in the PQ guidelines. Real users do this too. Results 
which give reviews and reputation information can be very helpful for a URL query. 

However, websites that offer usage statistics about a website are not usually helpful results for URL queries. Most 
users aren't interested in this kind of information. 



Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 




Women's Apparel & Fashion Accessories ! Coldwater Creek 

mvw coldwatercreek.com/ 

Coldwater Creek offers quality women's clothing in misses, petite and additional 
women's sizes. Shop online or at any of our stores for women's clothes and ... 


This LP is the target of the 
query. 










Lowest Medium Highest 


OT SR R U V 




[coldwatercreek.com], 
English (US) 


Coldwater Creek Coupons: Get $16 Off w/ Februan^ 2013 Coupon ... 

WiV.retailmenot.com > Clothing > IVomen's Clothing 

Discounts average $16 ofT with Coldwater Creels promo code or coupon. 53 Coldwater 
Creek coupons now on RetailMeNot. Get Presidents' Weekend Savings ... 


This LP displays coupon 
codes that users can use 
when making online 
purchases on the Coldwater 
Creek website. Online 
coupon codes are very 
popular in the U.S. 


Go to 






This is a well-known coupon 
website where users can 
often find working coupon 
codes. This result displays 
active coupon codes that 
would be very helpful for most 
users shopping online at 
coldwatercreek.com. 


httoV/www.coldwatercreek 
.com, the website of 
Coldwater Creek, a well- 
known U.S. company that 
sells women's clothing 
and accessories online 
and in retail stores. 


Lowest Medium HigfKst 


OT 3R R U V 




Coldwatercreek.com Reviews | SiteJabber Consumer Reviews of ... 

vn™. sitejabbercom/revi emsAnvw co Idwatercreekcom 
■kielfk A Rating: 5-3 reviews 

Is Coldwatercreek.com legit or a scam? 3 reviews "They are a great websitel It you 
check back often enough, they have great promotions and have some unique ... 


This LP has three reviews of 
the coldwatercreek.com 
website. 

This is a well-known review 
website. This result would be 
helpful for many users 
considering whether to make 
purchases at 

coldwatercreek.com. If there 
were more reviews and the 
reviews were more recent, 
the utility rating would be 
Useful only. 




Ldw^ Medium Highest 


OT 5R R U V 
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Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 


[ratemyprofGSSors.corn], 
English (US) 

Go to 

httpV/www.ratemyprofess 


Rate My Professors - Find and rate vour professor, campus and ... 
bl og.ratemyprofessors.com/ 

Over 1.5 million professors & 13 million opinions. Rate My Professors is the best 
professor ratings source based on student feedback. Find a rate a professorl 


This LP is the target of the 
query. 






Ltmest hAetfium hligh^ 


OT SR R U V 


Leslie Reece Scliichtel: RateMyProfessors.com Rankings Released 

m™.huffingtonpost comVleslie .r'CQllege-rankings_b_1 839051 .html 

Aug 31, 2012 - The rankings offer good news to both penny-pinching students and 

America at large. Less expensive alternatives to four-year Institutions are ... 


This LP is a Huffington Post 
article dated August 31 , 2012 
about the 

ratemyprofessors.com 
website. The article was 
newly published when it was 
added to the guidelines. 
Some users might be 
interested in this article. 






ors.com, a website where 
students can rate their 
college professors. 


( 1 1 1 1 1 

L:j\VE5t Medium Highest 


or SH R U V 


Towson's RateMyProfessor gets readv to test the marltet - Baitimo... 

Wivx. bizj ou rn a 1 s. c om./ba Iti m ore/stori es..'2 0 0 6..'-.i''story S htm 1 "J 

Apr 1 7. 2006 - Two Towson entrepreneurs bought RateMyProfessor.com last fall for 
seven figures. They're already looking for a buyer or some major investors ... 


This LP is an article dated 
April 17, 2006 about the 
ratemyprofessors.com 
website. Few or no users 
would be interested in this 
outdated information. 






Lmvest Medium Highest 


OT 3R R LF V 



24.0 Product Queries: Action (Do) vs. Information (Know) Intent 

Some product queries, such as [ipad reviews], have a clear information-seeking (Know) intent. Other product 
queries, such as [buy ipad], have a clear purchase (Do) intent. And some product queries, such as [ipad 
store.apple.com], have a clear navigation (Go) intent. However, most product queries don't obviously specify one type 
of intent. 

Even though the ultimate goal may be to purchase a product, many other activities may take place first: researching 
the product (reviews, technical specifications), understanding the options that are available (brands, models, pricing), 
viewing and considering various options (browsing), etc. 

Give high utility ratings to results that allow users to research, browse, and decide what to purchase. 
Important: PQ ratings for product results need extra care and attention. 

Often, the results for product queries are YMYL pages. Users need high quality information from authoritative sources 
when researching products, especially when products are expensive or represent a major investment/important life 
event (e.g., cars, washing machines, computers, wedding gifts, baby products). When buying products, users need 
websites they can trust: good reputation, extensive customer service support, etc. Results for product queries may be 
important for both your money and your life (YMYL)! 
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Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 


[high definition tv], English 
(US) 

This query doesn't specify 
a size or brand of TV. 
Users are probably doing 
research before making a 
purchase. 


HDTV Reviews j HDTVs Review j PCMag.com 

WArw.pcmag.com > ... > HDTVs & Home Theater 

PCMag provides up-to-date coverage and product reviews of HDTVs. 


This LP has comprehensive 
information about different 
types of high definition TVs 
(plasma, LCD, LED), as well 
as reviews about many 


Lowest Medijin Higlnest 


OT 3R R U V 


specific models. The website 
is known for having helpful 
reviews and information 
about electronics. 


[toyota prius], English 
(US) 

Most users who type this 

L|uc;iy cti |jiuuctuiy 

considering buying a 
Toyota Prius and want 
information on the car. 


2013 Toyota Prius 

WiW.edmunds.com > New Cars > Toyota > Prius 

Jan 2, 201 3 - Tlie 201 3 Toyota Prius continues to t>e ttie quinfessentiai hybrid t 
other newer hyljnds are more desiral5ie to drive. 


hThis LP is on a well-known 
car review website. It would 

Uc; Vc;l y llt:;l|jlUl lUl lllctlly Ul 

most users. 


Lo'iVESt Medium Highest 


OT 3R H U V 


[floral girl toddler jeans 
size 3T], English (US) 

The query is very specific. 
The user wants to browse 
and find an item to buy. 


Toddler Girl Jeans at babvGap | Gap - Free Shipping on S50 

mnv.gap.com > Home > Toddler Girls Clothing 

Our collection of toddler girl jeans at babyGap is fun. colorful and bright. Shop for a 
variety of fits and sizes offered in our toddler girls jeans collection. 


This is a helpful page for 
browsing and shopping for 
toddler girls jeans. There are 
many options for floral jeans 


1 1 1 h- 1 

Ls-WESt ME-diuiTif Highest 


OT SR R U V 


in size 3T. This is a high 
quality store known for selling 
jeans. 


[cufflinks], English (US) 

Most users probably want 
to find different cufflink 
styles and options to 
browse or purchase. 


Cuff Links for Men Nordstrom 

shop.nordstrom.com > Men > Accessories > Jewelry > Cuff Links 
Free shipping on cuff links at Nordstrom.corri:. 


This LP has a large collection 
of cufflinks from a very 
reputable retailer. There are 
many options and styles. 
Thi'? i'? a nnod nanp for 
browsing. There are helpful 
features at the top to choose 
styles, price ranges, etc. 


Low^ Medium hligh-est 


OT 3R R U V 



25.0 Mobile Landing Pages 

When rating BU tasks, you should assume that users are on a desktop computer or a laptop. Mobile rating projects 
have separate instructions. 

Most results in BU tasks are designed for desktop and laptop users. Please assign ratings to mobile-specific pages as 
follows: 

• When the mobile page is high functioning and would be satisfying for desktop/laptop users, then rate normally. 

• When the mobile page has limited functionality and a non-mobile version exists that would be much more 
helpful for users, please do not assign above SR. 

Please use your judgment when considering how helpful mobile versions of landing pages would be for users. Of 
course, if the mobile landing page is unrelated to the query or useless for any reason, a rating of OT is appropriate. 

PQ rating assumes that users are on a desktop or laptop and have a keyboard, mouse or track pad, and at least a 
normal sized monitor. Some mobile pages have very limited functionality and therefore are unsatisfying for a desktop 
or laptop user. A low PQ rating can be justified if the page is unsatisfying and/or does not achieve its purpose for 
laptop or desktop users. 
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Query and User Intent 


Result Block and LP, PQ Rating, Utility Rating 


Explanation 


[Stanford], English (US) 

Navigate to tlie 
liomepage of tlie 
Stanford University 
website. 


Stanford University 

wwA'.stanford-eclu/ 

Stanford University is one of theworid's ieading research and teaching institutions, it is 
iocated in Stanford, Caiifornia. 


The mobile version has 
less content and limited 
functionality compared to 
the desktop version. It 
would be helpful for few 
users. 






Lowest Medium Highest 


OT SR R U V 


Stanford Mobile 

m-Stanforcl.edu/ 

Stanford Mobiie. News; Events; Athletics: Maps Si Directions; Contact & Directory; Linl<s. 
Fuii Site. Events. Home ■ Featured ■ Today ■ Arts ■ Ciasses ■ Dance ... 






LuwKi Medium Hifltiest 


OT SR R U V 


[cbs sports], Englisli 
(US) 

Navigate to tlie 
liomepage of tlie CBS 
Sports site. 


Sports - CBSSports.com Sports News. Fantasv Scores. Sports Video 


Even though the mobile 
version has a lot of links, 
the font size is small and 
hard to read on a desktop. 
It would be helpful for few 
users. 


W)V. cbssports.com/ 

CBSSports.com features iive scoring and news for NFL footbaii. MLB basebaii, NBA 
basl<etbaii, NHL hockey, coiiege basl<etbaii and footbaii. CBSSports.com is ... 






LswESt Medium Higtiesi 


OT SR R U V 


CBS Home 
m.cbssports,CDm/ 

CBS Sports Home. Menu: Pistorius disputes murder claim, prosecutors say killing was 
planned. Today's Scores. Fantasy. Headlines. All Sports ... 






L-^west Medium Hjigtiest 


OT SR R U V 


[people], English (US) 

Navigate to the 
homepage of the 
People magazine 
website. 


People.com : Celebritv news, celebritv photos, and star stvie 


Although the versions are 
different, the mobile 
version has a lot of the 
same content. Some 
users might not even 
notice that it is a mobile 
version. This result would 
be helpful for many or 
most users. 


mv><v.pe ople.com/ 

Get your fx of the hottest breatang news, celebrity photos, fashion, videos and games 
from People Magazine. 


1 1 1 ' h- 1 

L[]-WESt Medium Highest 




OT SR R U V 


People Mobile - People.com 

www. people.com/people/mobile/home/ 

10 Best Celeb Quotes This Week. Photo Special. 1 0 Best Celeb Quotes This Week ■ 
Oscar Nominees: Before They Were Famous. Red Carpet. Oscar Nominees: ... 






LswEst Medium Higtiest 


OT SR R U V 




John Hinoklev. Jr - Wikipedia the free encvclopedia 


Although the versions are 
formatted differently, both 
LPs display the answer 
prominently and would be 


e n . wi ki pe d i a . org./wi ki.^J oh n_H i n ckl ey ,_J r. " 

John Warnock Hinckley Jr. (born May 29, 1955) attempted to assassinate U.S 


[guy who shot reagan], 
English (US) 






very helpful for most 


Lowest Medium Highest 


OT 3R R U V 


users. 


Find the name of the 
person who attempted 
to assassinate former 
U.S. President Ronald 


John Hincklev. Jr - Wikipedia. the free encvclopedia 


Note: This mobile page 
has all the same MC and 
SC and would be 
satisfying for 


e n . wi ki pe d i a . org.'wi ki.'J oh n_H i n ckl e V ,_J r. • 

John Warnock Hinckley Jr. (born May 29, 1955) attempted to assassinate ... The 
reason I'm going ahead with this attempt now is because 1 cannot wait any 


Reagan. 


desktop/laptop users. 




1 ' 1 ' 1 ' 1 ' 1 • 1 ' * ' 1 ' t 
OT 3R R U V 


Even though you have to 
click the arrows to show 
the MC, this LP would be 
helpful for most users. 


Ldw^ Medium Higtiest 
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Part 4: Using the Evaluation Platform 



26.0 Introduction 

Welcome to the Evaluation Platform! The Evaluation Platform is the system you will use to acquire and rate tasks. 



27.0 Accessing the Evaluation Platform (EP) 

Go to this link to access the Evaluation Platform: http://www.raterhub.com/evaluation/rater . 
28.0 Evaluation Platform Screenshot 

Here is a screenshot of the task acquisition page on the Evaluation Platform. Please note that changes are sometimes 
made to the appearance of pages on the EP, so it may not look exactly as it appears here. 




The red numbers represent the following: 

1. Tasks 

This text shows that you are at the task acquisition page on the Evaluation Platform. 

2. rater@qmail.com 

Your Gmail account. 

3. Recent tasks 

Click this link to revisit tasks completed in the last few minutes. 

4. Sign out 

Click this link to end your rating session. 

5. Acguire if available 

Click this link to acquire a new task. 

6. Experimental 

This text indicates that there is an Experimental task available. Please note there are other types of rating tasks 
(e.g., Result Review, Side-by-Side). Sometimes there will be only one task type available, but other times you 
may see more than one type displayed. When there is more than one type displayed, you may choose the type 
you want to acquire. 
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29.0 Block Utility Task Page Screenshot 



6 

I 



Evaluation ^latform 



rater@gmall.com I Rscpra tasks i sign nut 



Web I Experimental Average Estimated Time = 8.0 minutes 



.10 



Instruction 

Please refe^ to ttie General Cui(feline9 for I natruction on how to rate Block UEMHv laaks. 



Locale: Engllah^US) ^ 



12 



14 



T 



L1. Utility Rating {q,UQry-dependsiit)i 
■ li h I i I » M I M • I ■ M 



L2.| Utility bating {qusiynde pendent) 
■ k i I I I M > I M I I ■ M 



t3 ■ 



L3.r Utility Rating {qusiynde pendent) 



Pag« Qijality Rating i|query-inde pendent) ^1- Utility Rating (query-depsndenl) 



WA OT 



Same as RS 

YouTLiha Wikipflfiia. Ihe fraa snryrlnnflfiia 

VuiiliilMr i:: :■ vi[li!i] :>>i:iriii[| ivi:li;;]lii^. [:ii:;ilEMl Ijy Ihiiiir liinniii PMyM:!! H;riiF»l[]yi!i 
in rebnjary 2005. on wtiich users can upload. Miew and share videos. 
enwikipedia.org'wikiA'euTube 



T 

15 



Page i^usFity Rating {queiy-inde pendent) 



{r\ fSI Rfiprirl niifwi flonirmnl 

YouTUha for iPhons .IG.S, iPhons ^, iPhonfi AVi, iPhonfl iPnrf ... 

Dec ■i. 20^2 ... DBScnp4icHi. Walch the world's videes and keep up with yeur favorite YouTuba 

Chanrtel& with the ofTicia] VouTube appfor iOS. Sign i<n lo acoa&s ... 

ilunes a pple . comL'US'a.pp.'youtu be.i'id54400 7664? mt=S 



> I I 



> I 



Page i^usFity Rating {qMery-inde pendent) 
I I ■ I 1 k i I I 1 



Repon Dupe 



\Ei W \E 
YouTube - tJoutjIe 

YouTulw for Ardfoid. Milliors af videos at your finger! ip&. available on the ge 
wilh YauTube far Andrsid. Download YouTube for ^niiaid^ tial an Android? 



:iii<]l4- 



L^J Utility Rating {quefy-dependent) 



P I' 



I I I 



» I > I ■ 



' I 



Page Qijality Rating i|query-inde pendent) 



I 



I 



I 



Page Qijality Rating i|query-inde pendent) 



[Pl riSl ^n^'"* I'"!"- 

YouTube - Android Adds on Google Play 

OvC 11, 1^01? ... I>i|! kitlEsI vi.ii::inn al Ihui iiHic^iid Vciii liitm :ipp K^-iFp iiii witli yai 
Vi^uTulW i:JlWn4l$ !lrKl ni^Cnsn the wCilcT^ ifliAm!^ iinywIiEHii vnLFi Lhin ... 
play.gaogrile.conv'slore^apps/details7»d=i:am.go[]gle.andrajd.youtube&hl=en 

Utility Rating {query-dependent) 
I M I 1 I h I M M M ■ M I 

duk .-^T a L« u 

H (E [H ^''V<' 

YauTUha — Gr^oqSfl Dnvfl'linpA'r^ 

Ditc 19. ?ai? ... liiLiN]i:itiE ViiiiliilK- viilcMj:; I 
de\^1ci pers .geogle. conVyoutu be/ 

I 



Utilily Rating (query-^ependenlj 
I 4 I M 4 I I H 1 M I 1 I 1 



Page Qualily Rating (query-independem) 



H/A OT 



Dupe ofin? 
YouTuba 



v/vrtv, yo ut u be . c o riv' 



Page Quality Rating (query-independent) 



wsvw youtu be . corrv'chann eh'HCp-Rdqh 3z4Uc 

Country - Pop Music - Call Me Maybe - Rock - YauTube 10<D 



TO. laiilily K;iliii[| (ciiJirrv iii^Fx^rinJi^ntk 

■ j,-'''i'"y"i"'i 

EES 

Uupe oTKI 
You Tube 

wwiv, youtu be . coriv' 

I'll :-.i<ir v«]i.iT H:irli:Fii SM^ikii ;iriHJ tii\r.i\ y 
auf entire office diaing it. reddil: the 

Utility Rating (query-depend en I) 
I i ' 1 M " I M «^Mt=*=1=*=1 

t*A or Sfl H u V 



2:47 231 likes. 7dislikes. by 

L il]i'rlyMu::ii:CJr[>iJn ^'IU.S.FiEI vii:vu:; 

YouTube ... 

Utility Rating (query-dependent) 



Miiiciii {jiirJily K;iliii[| (cfui^ry iniJiriiHirKjitril;) 



Page Quality Rating (query-independent) 
I h I I > I > 1 < I 



Page Quality Rating (query-independent) 



iE] m \m ^''3""^ ''"F"; 

Same as LI 

YouTube - WikipoiJia. tiio tree oncvclopoiJia 

Yciiitiitrc! iii viiJii-E] lih^irmci vui^tKiiti:. E:rii!'jjti:[E t>y LhirL>4! turiiii^i MiiyP^il 
in Fi-I]ru;iry PQOfi. an virtiii;;^ ii^i^-r:: i;:;iri upl»rid. viuw .jnidi ;;l!i:ji4i utdicD:: 
en .wikipBdia. org^'wikii^ouTu be 



16 ■ 



I I I confirm tliat thare are na addlttoriiH;! dLrplicata reaults abcva 



t 

17 



Submit and Stop Ftating 



t 

18 



t 

19 



The red numbers represent the following: 

1. rater@qmail.com 

Your Gmail account. 

2. Recent tasks 

Click tinis link to revisit tasks completed in the last few minutes. 

3. Sign out 

Click this link to end your rating session. 
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4. Web 

This text indicates tinat tine current task belongs to tine Web property type. Otiner properties you may rate include 
Video, Image, Mobile, etc. 

5. Experimental 

This text indicates that the current task belongs to the Experimental project type. Other project types you may rate 
include Result Review, Side-by-Side, etc. 

6. Average Estimated Time 

This text indicates the average estimated time for completion of the task. 

7. Instructions 

Click this tab to view the project-specific instructions. Some tasks display project-specific instructions on the task 
page instead of behind the Instructions tab. 

8. Query 

You should understand the query before rating the task. 

9. Locale 

This refers to the task location and task language associated with the query. 

10. Report a Problem / Release this Task 

Click this link to report a problem and/or release the task. 

11. Utility Rating Slider 

Use this rating slider to assign a utility rating. 

12. Page Quality Rating Slider 

Use this rating slider to assign a page quality rating. 

13. P - F - D Flags 

Assign any of these flags (Porn, Foreign Language, Didn't Load) that apply to the result. 

14. Report Dupe 

Use this button to report duplicate results. Results are considered to be duplicates if they have the same landing 
page. For more information about reporting dupes, please see Section 32.0 . 

15. Comment 

Use the result block comment box to write comments that will be helpful to you when assigning utility and PQ 
ratings. Note: All comments must be written in English. 

16. Dupe Confirmation Checkbox 

Check this box to confirm that you are done flagging duplicate results. 

17. Submit 

Use this button to submit the task and continue rating. 

18. Submit and Stop Rating 

Use this button to submit the task and stop rating. 

19. Cancel 

Use this button to cancel any ratings you have assigned in the task. 
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30.0 Notes about Using the Block Utility Rating Interface 



Result blocks are sometimes slow to load: Before assigning any ratings, please scan tine task page to make sure 
tinat all result blocks with a utility rating slider have loaded. Please note that result blocks may be slow to load and/or 
that the task page may need reloading if there are empty result blocks. If the result blocks fail to load after refreshing a 
few times, please release the task. 

Sometimes a side will not have any result blocks: If one side displays the message "This side did not generate any 
results," please do not release the task. 

Do not refresh the task page after assigning ratings: If you refresh the page after assigning ratings, you will lose 
them and they will have to be entered again. 

31.0 Using the "Report a Problem / Release this Task" Button 

You will report problems and release tasks using the "Report a Problem / Release this Task" button on the rating 
interface. This is what the "Report a Problem / Release this Task" button looks like: 



Report a Proble^m t Release this Task 



Clicking the red button displays the following reporting and release options: 



Report a Problem / Release this Task @ 




Select One 

O 1 lack expertise for this task. 

© 1 did not sign up for adult content. 

©The instructions are not clear. 

© The task is in the wrong language. 

© The average allocated time is not SLifficient. 

©There is a technical problem with this task (please describe). 

O Other (please describe) 


Comment 




A 


1 


Release this task? 

® Yes - Give me a new task. 

© No - Send my feedback but let me continue to work on this task. 


Will a Bcreenshot help us understand the issue? 

© Yes - Please take a screenshot for me. 
® No - A screenshot is not needed. 




Submit neport | 
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Here are some examples for the "lack expertise" and "technical problem" release categories. 
Lack expertise: These queries require specialized knowledge and may be difficult for some raters. 



Query 


General Query Topic 


[Cortisol secretion shows a blank pattern of secretion] 


Scientific query 


[minecraft gun and vehicles 1 .1 mod installer] 


Gaming query 


[python pandas index like matrix] 


Technical query 



Technical problem: These result blocks have question marks and other characters where text should be. 



L1. Utility Rating (query-dependent) 



E El H 

Samp a-B PI 



Report Dupe 



I -1X1 a U 



I -inSO ,^gP ^°021-68361966 18918321661 www. - M^sa-j 



- ,,a[0 ^"021-68861966 _ 



isyiBj^dibbi www iian.xiay.CDm - ffiiii 
wtw.dearKay.com/qcl_mvw.1ianxiay.com.html 



Page Quaiity Rating (query-independent) 
I I ' I ' I ' I ' I 

NiA Lmast k'esiLrr Hsr«t 

Comment 



I 



Utiiity Rating (query-dependent) 

1 I AU-iL.^-ik,L.k,i[..k.i..i.[„iL i 
N/A OT SR R U V 

[g [g [g Report Dupe Comment 




1 T?7 77 7??77?? 7777? 77777 - ?|/indows 7 ultimate 777 






...... 




A 











I 



Note: Based on the number and/or type of tasks that you release, you may be asked to provide additional details about 
the reason for some of the releases. 



32.0 Reporting Results with Duplicate Landing Pages 

In addition to assigning utility ratings to each result block, we would like you to tell us whether any results are 
duplicates (dupes). 

Pre-ldentified Duplicates 

Some duplicate results are automatically detected and will be pre-identified for you. They will be highlighted by a grey 
or green background, and annotated by either "Same as..." or "Dupe of..." text right below the slider and above the 
result block. 
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This is winat tinese pre-identified dupes look like: 



Same as... pre-identified duplicate result 



R3. Utility Rating [query-dependent) Page Quality Rating [query-independent) 

I I M I I ' I I I ' M I ' M I I I ' I' ' I ' I ' I 

N/A OT SR R Lf V HA Lowest Medium Highest 

[p] [f] [d] Report Dupe Comment 
Same as L6 ^ 

Facebook - Wikipedia , the free encyclopedia 

History ■ Company ■ Website ■ Reception 

Facebook [stylized facebook) is a social network service and website launched in February 
2004 that is operated and privately owned by Facebook, Inc. As of January 2011 ... 
en.wikipedia.org/wiki/Facebook 



Dupe of... pre-identified duplicate result 



L1. Utility Rating [query-dependent) Page Quality Rating (query-independent) 

I I ' I I I ' I I I ' I I I ' M I I I I ^ ' I ' I 

H'A OT SR R U V WA L^WEit IWediLm Highest 

[p] [f] [d] Report Dupe Comment 

□ upe of R1 

Welcome to Facebook - Log In. Sign Up or Learn More 

Facebook is a social utility that connects people with friends and others who VKork, study and 
live around them. People use Facebook to keep up with friends, ... 
www.facebook.com/- Cached - £-;imilai' 

Login Create a Page 

Find Friends 0 PetVille 

More results from facebook.com » 



Please note: 



• Results annotated by "Dupe of..." may be given different ratings/flags/comments. 

• Results annotated by "Same as..." may not be given different ratings/flags/comments. Their utility ratings, PQ 
ratings, flags, and comments will be automatically transferred to each other. 

• You cannot un-check dupes that have been automatically detected and pre-identified. 

Rater-Identified Duplicates 

We would like your help identifying duplicate results that have not been automatically detected. Please mark two 
results as dupes if they have essentially the same content on the main landing page AND you would not want a search 
engine to return both results for the query. 

Please note that in this project dupe identification is query-dependent. 

• Specific queries: For queries where the user is looking for a specific piece of content (such as queries looking 
for song lyrics, queries looking for a specific article, etc.), obtaining that piece of content from different sites 
could be helpful for users to verify the information, so they should not be rated as dupes. 

• Broad queries: If the query is broad, then returning the same piece of content is not what the user is looking 
for, and hence those results should be flagged as dupes. Results may be considered dupes even if they have 
different minor content on the page (such as different ads, images, or related links). 

Please identify dupes both within the same side and across sides. Even for cross-side results, you should still ask 
yourself the question, "Would you want to see both results if they were returned by the same search engine?" 
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Reporting Duplicate Results 

When you notice that the results in two or more result blocks are duplicates, please click on the Report Dupe button of 
one of the results. The current result (the result you clicked on) will be highlighted by a thick solid red border and the 
button's name will change to Select Dupes. 



L2 Utility Rating (query-dependent) 
g I I I I I I I I I I I I I I I I I 



Page Quality Rating (query-independent) 
I I ' f ' [ MM 

WA Low-eat Med i um Highest 



EES 
Login I Facebook 

Facebook is a social utility that connects people with friends and others ... 
Sign Up for Facebook - Touch Facebook - Facebook | Facebook - Mafia Wars 
www.facebook.com/login.php - Cached - Similar 



Comment 



You can then check results that are duplicates of the current result, and the checked results will be highlighted by a red 
dotted border. 



3-42. Utility Rating (query-dependent) Page Quality Rating (query-independent) 

I I ' I ' I ' I I I ' I I I ' I ' <l I I ' I ' I ' I ' I 

i M'A OT SR R U V rj.'A LowEst Medium Highest 

i [F] [f] [d] 12 Dupe of L2 Comm,ent 
i Dupe of L2 

Login I Facebook 

: Facebook is a social utility that connects people with friends and others who work, study and 
: live around them. People use Facebook to keep up with friends, upload an unlimited ... 
iFacebook. com/home, php 



The duplicate results that you checked will be annotated by "Dupe of..." text right below the sliding rating scale above 
the result block. After selecting all dupes, please click the red Select Dupes button to return to the normal rating 
mode. The button's name will change back to Report Dupe, and you will be able to report other sets of dupes, if there 
are any. If you change your mind, you can always un-check a result. 



1-2. Utility Rating (query-dependent) Page Quality Rating (query-independent) 

I I ' I I I ' I I I ' I I I ' I >J| I I ' I ' I ' I ' I 

I (i|/A OT SR R U V H'A LawKt IWacJium HighefSt 

[p] [f] [d] Report Dupe Comment 

Dupe of R2 

Login I Facebook 

Facebook is a social utility that connects people with friends and others ... 
Sign Up for Facebook - Touch Facebook - Facebook | Facebook - Mafia Wais 
www.facebook.com/login.php - Cached - Similai' 



n 
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Dupes 

QUERY: [choosing and installing a motorcycle battery] 

URL 1 : http://www.caimaq.com/wordpress/201 0/03/06/motorcvcle-battery-how-to-choose-install 
URL 2: http://www.articlesbase.com/motorcycles-articles/choosing-and-installinq-a-motorcvcle-battery-47798.html 
Reason: Both of these results display the same article (which also appears on many other pages on the web). The 
only real difference between the landing pages are the Ads displayed around the article. The query is broad enough 
that users would not benefit by search engines returning more than one of these results. 

Dupes 

QUERY: [jason castro] 

URL 1 : http://www.myspace.com/iasoncastromusic 

URL 2: http://www.myspace.com/iasoncastromusic?MyToken=503599bf-01cf-4427-bdf4-d63920c107f9 

Reason: These two results have the same landing page, even though the URLs are different. Users would not benefit 

by search engines returning both results. 

Not Dupes 

QUERY: [material girl lyrics] 

URL 1 : http://www.lvricsfreak.eom/m/madonna/material-Hqirl 20086925.html 

URL 2: http://www.lvrics007.com/Madonna%20Lvrics/Material%20Girl%20Lvrics.html 

Reason: Even though both pages display the lyrics to the song "Material Girl," users would probably want to have the 
option to visit both pages so that they could verify the accuracy of the lyrics. Users could benefit by search engines 
returning more than one page with the lyrics to the song. 



33.0 Using the Page Quality Rating Scale in Block Utility Rating Tasks 

You will use the Page Quality slider to rate the PQ of the main landing page of the result block. 

Important: Due to the number of results in most block utility tasks, you will base your PQ rating on the landing page 
questions alone (e.g., identifying the purpose of the page, identifying the MO, SC, and Ads, rating the quantity of MO, 
etc.). You are NQT required to perform the website level checks required in PQ rating tasks. In other words, you are 
not required to learn about the reputation of the website, assess whether the website is maintained, etc. However, if 
you do feel that the landing page seems low quality and you want to explore the website for corroborating evidence, 
please do so. 

Reminder: Most results in block utility tasks have both utility and PQ rating sliders, but some results only have a utility 
slider. 
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